*ren* PSN Down, Customer Info Compromised

[BoardBonobo]

Every day big name corporations, banks, and even governmental departments are hacked into. They lose money, data, any number of things. Sometimes they get stuff back, sometimes they don't. In some cases they are even held to ransom over access to their own systems. I once stood in a room with a forensics team and watched money literally disappear from the ultra secure financial system that held it, within in seconds it was all over the world. I think in that instance they recovered about 45% of what they lost. The rest was paid for by insurance and then their customers. Shareholders, as always, were the last to lose out.

The point is this thing happens all the time. It just doesn't pay to have the general public living in a constant state paranoia about the systems that organise and enable their lives. If the general population were to ever lose faith in any particular system, to the point of refusing to use it, can you imagine the chaos that would ensue?

Sony have been publicly humiliated by the general disclosure of their particular form of security breach. If this was the primary goal of the hackers then they have done an admirable job. If they want people to think that Sony is a special case then the reportage of this case is doing a great job. At the end of the day that just takes the heat off everybody else who, you can be damn sure, are now quietly doing some pretty in depth security audits.

And the stolen data itself is not the problem. That lies with the systems that it can be used to subvert. Our payment systems are woefully inadequate, electronic transfer is open to this kind of abuse. In many ways they are still paper based systems that have have been bolted onto what is now the cloud. Those are the systems that need to change. This kind of data leak will be with us for the foreseeable future, the underlying systems will ensure that. What needs to change is how effective such a leak will be at enabling crime and how quickly they can be detected in the outside world.

 

[deathindustrial]

For Samsung I would assume that *is* the case just because they are a hardware manufacturer foremost though I have not spent much time researching it.

For Facebook, that is how they work by design, it is intended to give away personal details. Even with that in mind, they have had numerous security issues over the years. . .here's some recent ones:

http://www.theregister.co.uk/2011/04/13/facebook_fixes_hotmail_reset_bug/
http://blog.trendmicro.com/facebook-stalker-tracker-tool-turns-users-into-spammers
http://www.theregister.co.uk/2011/05/10/facebook_user_credentials_leaked/

Google was breached somewhat recently though to my knowledge they never fully clarified what all was accessed:

http://www.wired.com/threatlevel/2010/01/operation-aurora/

They also have fairly regularly have XSS and similar issues creep up with their various web properties as well as malicious ads popping up via adsense.

For Amazon, there was an old school hack a decade back:

http://www.theregister.co.uk/2001/03/07/amazon_despite_denials_was_warned/

Not sure about more recent things beyond their cloud services getting used by criminals for cheap cycles.

For MS, they have had large issues in the past similar to Sony:

http://www.theregister.co.uk/1999/08/30/massive_security_breach_affects_hotmail/

I am assuming you are not lumping in their horrendous application security history here though perhaps IIS and Exchange deserve to be since they are what have often led to breaches for other entities.

MS, Google and Facebook have all silently fixed server side issues in the past so again so I do not think it is entirely unreasonable to expect that breaches have gone unreported for these companies. This will be very common as well with smaller companies, especially those using shared hosting for e-commerce apps. There is almost zero enforcement and lots of negatives to admitting to breaches so if they can avoid it, they will.

Cheers

 

[patsu]

Bugs are fixed and patches are applied as part of maintenance cycle. If there is no user impact (info leak, $$ lost, service downtime), then there is no need to go public.

I think online game servers get attacked frequently because people are curious or want to cheat. ^_^

 

[Shifty Geezer]

For Samsung...

You missed the point. . You said how could we know if Sony wasn't hacked earlier then was reported. This is true of any and every organisation. You only know about the hacks that are found and reported. If one is willing to consider the possibility that Sony have been leaking info for far longer than this, one has to entertain as equally valid the possibility that other companies are leaking info. Whether any of these companies has been reported as having security breaches doesn't affect the hypothetical "what if" you presented. A company that has never reported a security breach would come across as having great security, but it could be they either never spot the breaches, or they hush them up for PR purposes. You never really know.

 

[3dilettante]

Sure. Based on the situation it may be appropriate. But, according to their timeline, this is the first thing they did. To me, this indicates that at that time they had inadequate internal resources.

Whether Sony had the resources or not, it is a potential conflict of interest to have the same people who set up a system that was compromised review it for security problems. It is also a security risk if you can't be sure there wasn't internal involvement.

Outside observers, regulators, and clients are also not likely to be convinced by a self-assessment that everything is fine.
Some comments I've seen indicate this is not out of the ordinary for a company that experiences such a data breach.

 

[mrcorbo]

Whether Sony had the resources or not, it is a potential conflict of interest to have the same people who set up a system that was compromised review it for security problems. It is also a security risk if you can't be sure there wasn't internal involvement.

Outside observers, regulators, and clients are also not likely to be convinced by a self-assessment that everything is fine.
Some comments I've seen indicate this is not out of the ordinary for a company that experiences such a data breach.

I don't think that makes sense in this situation. If you're doing a review of your security as a preventative measure or as a post mortem to an event like this, yes. But in the midst of a crisis, you want people in-house with the overall knowledge and more importantly the specific knowledge of your systems and operations that can begin to act immediately.

 

[patsu]

I believe they acted immediately with internal resources first. If I remember correctly, one of the articles mentioned that they brought in 2 groups of security consultants (The first consultants recommended them to bring in extra help). I doubt the internal team simply stood by and watched. They also need to investigate and re-setup the compromised environment in parallel. Someone has to do the work.

 

[mrcorbo]

I believe they acted immediately with internal resources first. If I remember correctly, one of the articles mentioned that they brought in 2 groups of security consultants (The first consultants recommended them to bring in extra help). I doubt the internal team simply stood by and watched. They also need to investigate and re-setup the compromised environment in parallel. Someone has to do the work.

Yes, they acted immediately. It just appears that those actions consisted of "flip the switch and call for help". I'm sure the congressional inquiries will clear this up, though.

 

[Shifty Geezer]

Yes, they acted immediately. It just appears that those actions consisted of "flip the switch and call for help".

That's unfair. They would have looked at the data, made choices, tried to work out what was going on, and then when they realised it was too big a job, called in help. Like anyone. You can't expect a company to keep on retainer a body of expensive experts who's usefulness would only be needed once every five to ten years when there's a major new attack! All companies outsource as needed, and this is no different to, say, a bank calling on a security firm to transport money instead of establishing its own security services. Or a mail-order company employing external postal firms instead of establishing its own delivery infrastructure. Full forensic analysis isn't anything you'd want as a full-time part of your commercial organisation.

 

[-tkf-]

Yes, they acted immediately. It just appears that those actions consisted of "flip the switch and call for help". I'm sure the congressional inquiries will clear this up, though.

Ehmm, we had a lengthy discussion about this and you post this when you know the story was different?

 

[-tkf-]

Full forensic analysis isn't anything you'd want as a full-time part of your commercial organisation.

But you would expect a security team which by all accounts could have reacted faster. Of course, we don´t know to what extent there is anyone responsible for security within the PSN team.

 

[Shifty Geezer]

There's different degrees of security though. You wouldn't expect every local law-enforcement agency to have a full SWAT team on hand - they'd have whatever level of security was appropriate for normal activities, and call in reinforcements as needed. We don't know the nature of the attack, despite those who are very quick to say it was nothing other than complacency and bare-minimum security on Sony's part. It's quite possible this was an attack no-one expected via a new vector, maybe even an employee traitor, and as such there was no realistic way anyone could defend against it. Yes, having a troop of the world's best security experts on hand 24/7 patrolling their servers would have given improved security, but at a potentially ridiculous cost. Life is a matter of compromises, always. There's no situation where you can forgo all compromise and buy the very best - there'll always be more you could do by spending more money. Whether Sony's compromise erred on the side of cheapness or reasonable standard or moderate effectiveness or insanely good, we don't know. Sony have said that their people looked into it, and some are just second-guessing what level of standard those people are.

 

[BoardBonobo]

But you would expect a security team which by all accounts could have reacted faster. Of course, we don´t know to what extent there is anyone responsible for security within the PSN team.

But even a security team would have a mammoth task on their hands. You've noticed something fishy in a log, or a cron job has been interrupted, or maybe your database has reported an unusual amount of transactions. Whatever is was that alerted you in the first place you've got track the source. Is it just a script gone wild, unusual behaviour due to a race condition some code somewhere. Once you've looked at that on your server and you realise that it's actually an intrusion.

Next visit is to the log files, firewall, syslog, access and error logs. Now any of these can be hundreds of megs in size literally millions of entries. And they've all got to be gone through with a fine tooth comb to find the precise point of entry. You don't want to reset the server as the intrusion code might just be a virtual device which will self destruct if you do e.g. /dev/shm.

Going through the logs you start to realise that the intrusion has gone deeper than anything before and it's starting to look like the hacker(s) might have got close having low level access. That means they may actually have stolen the log in details of any staff member with access to server systems. Now you have to change every password on the system. Now you are looking at logs for systems that may very sensitive to the business itself. It's a complete nightmare.

This is the point you sever outside access and call in the big boys to do the rest. All of this can take days for a single server, if you're looking at dozens if not hundreds of servers that may have, potentially, been compromised then the task just grows exponentially.

No matter how fast you are, or how good your systems are at detecting unusual activity it all takes time to do the actual sleuthing.

 

[mrcorbo]

Ehmm, we had a lengthy discussion about this and you post this when you know the story was different?

It's an oversimplification. I'll grant you that. As you said, it's covered ground and I didn't particularly want to go over the whole thing again point by point. Especially when all I was trying to get at was..

But you would expect a security team which by all accounts could have reacted faster. Of course, we don´t know to what extent there is anyone responsible for security within the PSN team.

..pretty much that.

There's different degrees of security though. You wouldn't expect every local law-enforcement agency to have a full SWAT team on hand - they'd have whatever level of security was appropriate for normal activities, and call in reinforcements as needed. We don't know the nature of the attack, despite those who are very quick to say it was nothing other than complacency and bare-minimum security on Sony's part. It's quite possible this was an attack no-one expected via a new vector, maybe even an employee traitor, and as such there was no realistic way anyone could defend against it. Yes, having a troop of the world's best security experts on hand 24/7 patrolling their servers would have given improved security, but at a potentially ridiculous cost. Life is a matter of compromises, always. There's no situation where you can forgo all compromise and buy the very best - there'll always be more you could do by spending more money. Whether Sony's compromise erred on the side of cheapness or reasonable standard or moderate effectiveness or insanely good, we don't know. Sony have said that their people looked into it, and some are just second-guessing what level of standard those people are.

I'm comfortable enough with what I do know to make some inferences. There would have to be some pretty amazing twists in this story for the pieces not to fit the way I expect they do. Add in the broader context of how Sony have performed as a company both financially and technically as they have built out these systems and the teams that run them and the schizophrenic nature of their management (which you both have been lamenting recently) and I find it hard to be persuaded by arguments advocating that I should give them the benefit of the doubt.

 

[BRiT]

No matter how fast you are, or how good your systems are at detecting unusual activity it all takes time to do the actual sleuthing.

Yes, but once you decide it's so bad that you have to effectively pull the plug on the service(s), you should at least notify your customers and not delay for another week before doing so. That action there is what makes Sony look incompetent in the eyes of some consumers.

 

[patsu]

But you would expect a security team which by all accounts could have reacted faster. Of course, we don´t know to what extent there is anyone responsible for security within the PSN team.

React faster compared to what ? Do we know how many attackers were there ? (I have no idea)
Shutting down the entire PSN isn't a small decision. One of the official blog posts mentioned that they had to pull the plug when more and more machines become suspect.

Yes, but once you decide it's so bad that you have to effectively pull the plug on the service(s), you should at least notify your customers and not delay for another week before doing so. That action there is what makes Sony look incompetent in the eyes of some consumers.

Not just that. The initial communication was also not so accurate. One of the posts mentioned "password" instead of "password hash".

...

Also interesting is the DoS attack shortly before the breach. I wonder if the attackers took the opportunity to compromise the service while everyone was distracted.

 

[BoardBonobo]

Also interesting is the DoS attack shortly before the breach. I wonder if the attackers took the opportunity to compromise the service while everyone was distracted.

I would suspect, based on my own experience, it was whilst they were shoring up the firewall rules to cope with the DDoS attack from the EC2 cloud that someone noticed something awry in the firewall logs. Unusual traffic on a specific port perhaps.

 

[Shifty Geezer]

I'm comfortable enough with what I do know to make some inferences.

Where does your knowledge come from? Unless you have sources other than the same media outlets spouting nonsense that well all have had to put up with, you have next to no information. It's your prerogative to blame first and decide Sony are guilty until they can prove themselves innocent in your eyes, but no-one who wants to make a fair judgement will act on anything less than pretty concrete info. BoardBonobo has explained in detail how these things can work out. There's even been suggestion of a disgruntled employee being 'let go' taking revenge. We don't know anything really about the ins and outs, and yet you'll make a judgement call because you are comfortable with what you think you know.

Yes, but once you decide it's so bad that you have to effectively pull the plug on the service(s), you should at least notify your customers and not delay for another week before doing so. That action there is what makes Sony look incompetent in the eyes of some consumers.

That's a valid point, but BoardBonobo's reply was specifically aimed at those who feel Sony shouldn't have needed outside help and should have been able to sort things out faster than they did (or are doing, Store still being down). Hopefully these people now see how complex the situation can be, and how their ideas of how things should be run aren't realistic.

 

[goonergaz]

One thing that concerns me, that I've mentioned before, was the need for Sony to bring in outside firms to investigate the intrusion. Ideally, wouldn't you want to have a sufficient level of expertise in-house to deal with something like this? The fact that they created the new position of CISO after the intrusion happened points to them realizing that they had a deficiency before. Not recognizing the need for this position beforehand points to a lack of competence in their management, at least in this specific case. It further leads me to suspect, that their staffing below the management level was probably insufficient as well for the operation they were being asked to oversee.

This to me is a rediculous assumption - the overheads would be astronimical for something that had never happened. I'm sure Sony had a level of 'expertise' but the breach was potentially very bad so brought in unbiased experts to give a full account of the damage so they know the worst case (ie staff won't be able to cover up or only tell half the story). Certainly I can't think of any company that doesn't use a form of 3rd party support...even Microsoft.

No company has 100% cover for every scenario - especially in the current climate where companies are cutting what's seen as 'fat' - even where I work bizzare descisions seem to be made and good knowledge seems to be made redundant - alternatively maybe they just didn't replace a person who had recently left, who knows - but the point is the same, no company has every angle covered no matter how important it is - every company/person improves aspects after bad things happen, unfortunately this was a very bad thing.

Look at airport security - it's fair to say that there's more than enough evidence to prove my comments are valid - and in those cases we are talking peoples lives not data!

 

[mrcorbo]

Where does your knowledge come from? Unless you have sources other than the same media outlets spouting nonsense that well all have had to put up with, you have next to no information. It's your prerogative to blame first and decide Sony are guilty until they can prove themselves innocent in your eyes, but no-one who wants to make a fair judgement will act on anything less than pretty concrete info. BoardBonobo has explained in detail how these things can work out. There's even been suggestion of a disgruntled employee being 'let go' taking revenge. We don't know anything really about the ins and outs, and yet you'll make a judgement call because you are comfortable with what you think you know.

Sony and yes.

That's a valid point, but BoardBonobo's reply was specifically aimed at those who feel Sony shouldn't have needed outside help and should have been able to sort things out faster than they did (or are doing, Store still being down). Hopefully these people now see how complex the situation can be, and how their ideas of how things should be run aren't realistic.

To be clear, I am not saying that they shouldn't ever need to call for help. I am saying that they should have had the internal resources to accomplish more on their own. Especially when it came to keeping their customers informed.