Rant mode: I hate utter morons who work in IT

[Albuquerque]

The organization that employs me is a bit small, perhaps 275 employees by the end of this year. We are a not-for-profit medicare provider for a small region in the state of Kentucky. My staff is a total of five -- three service desk folk, and two infrastructure folk.

Most of the employees here use a data entry system based in a hosted Citrix system provided by a much larger organization (a well-recognized name in the medical data field, but who shall remain nameless) based in another state on the eastern seaboard.

They are migrating their website to a new URL, and a month ago informed me that the "old" URL would stop functioning soon and that my users needed to change their bookmarks and links. By itself, that's a fair ask and something that my team would certainly handle.

Except, their new URL has a broken SSL certificate. When I visit the new URL, the browser warns that the certificate FQDN does not match the URL that was entered. Now, as an IT "guy", I get what is broken and I understand that it's probably just an oversight on their part. A glaring oversight that should've been caught on their very first attempt to use the site, but whatever.

I send them a note: "Hey, your cert is b0rked. I really can't migrate users to a broken SSL site until you get that fixed. Lemme know when you're done."

Nothing. For three weeks. I finally sent another note at the beginning of this week, after noticing they still haven't fixed it, saying something akin to (but not verbatim of course): "Hey, uh, so this is still just as broken today as it was three weeks ago. Any ETA on this so that we can get moving?"

The response I get, distilled into one sentence: "What browser are you using?"

Wait, what the fuck question is that? I reply back, with the obvious: "No, you dolt. This isn't a browser error, this is YOUR CERTIFICATE IS BROKEN. Go fix your shit."

They want a screen capture, in IE, because they think it's a ??? browser issue ???.

Now I'm angry. I send them a screencap in IE8, IE9, IE10, the IE11 beta, from Chrome, from FireFox, from Opera, from Mozilla, from Safari, and even took a screencap from my android phone on the AT&T network -- all of them showing the certificate failure.

Their response,distilled into yet another single sentence: "Oh, yeah just like CONTINUE and it will be fine."

This is a system that houses personally identifiable health information, which is absolutely under the strict guidelines specifically outlined in HIPAA law. Our only source of income is federal and state funding for Medicare. And your'e telling me that I should teach 200 of my users to IGNORE A CERTIFICATE FAILURE that specifically says "THIS ISN"T THE WEBSITE YOU THINK IT IS", and click Continue, and it will be fine?!?

WHAT.

THE.

FUCK.

I sent back the note simply saying "No, I will not accept that solution. You will fix your SSL certificate for your website, or my legal team will talk to your legal team about breach of contract."

It's a damned SSL cert, I just bought a wildcard cert for my company's domain name for $700 for five years. Quit being cheapasses and FIX YOUR SHIT.

</rant>

 

[Grall]

I don't really understand why you don't call up some bigwig at your IT provider's main office and have a serious discussion about the incompetence of the staff working in their tech department.

To even suggest to train people to ignore certificate errors when dealing with sensitive information is...unforgivable, in this day and age. It's like they haven't even heard what kind of damage social engineering and (spear)phishing etc can cause.

To not get something like this fixed for weeks is just incomprehensible. Certificates exist for a reason.

 

[Albuquerque]

I can't even begin to convey how many times these people have royally fucked something IT-related. Their service desk is atrocious, tickets will be submitted for a user to be created, and it will not be completed for three weeks (with a five day SLA.) Or a ticket will be submitted to have access modified, only to result in access being fully removed. Or a ticket will be submitted to disable a termed employee, only to have that employee show up on an active user review audit six months later.

Projects for system upgrades that fail massively and for months on end. Projects to move us to a two-factor authentication system that result in simply giving everyone a static "password token" rather than actually getting the little key-fobbie things.

The root cause of these problems? They're our former parent company, and we have some absolutely horrible contract with them that predates anyone seeming to "care" about service level agreements, security, and even (apparently?) basic functionality of the product that we're paying out-the-nose for.

Every job has it's epic downside, this is mine.

 

[itsmydamnation]

browser can matter . why?

some browsers user there own certificates store and certificate chaining can be a bitch.

I have had to custom write scripts for load balancers that will look at accessed http content to determine the web browser used and then deliver a different certificate chain for different web browsers.

This was with a Verisign cert so its not like it was a dinky little cert.

 

[Albuquerque]

I agree that a browser can have implications to cert revocation, but this topic here is nothing like that. The cert itself is for a different domain name than the DNS record that points to the server hosting that cert.

It's as if I typed into my browser: https://www.awesomesauce.com

But then my broswer indicates the certificate is for: https://www.failsauce.com

This specific error can never be a browser failure, it's a failure of the IT staff to put the right certificate on the server.

 

[homerdog]

That should be a HIPAA violation, so report them.

I also work in IT and can vouch that many people who work in IT are total morons who know nothing about IT.

I think the root of the problem is that most HR folks know nothing about IT, making it very difficult for them to hire good IT personnel. The ones who are best at it let the IT staff hire the IT staff.

 

[Mize]

It's probably your browser. Have you tried IE?

 

[BRiT]

Have you tried turning it off and on again?

 

[tongue_of_colicab]

Have you tried turning it off and on again?

You'd be surprised how useful this is in some cases I work in IT related to commercial shipping and when something goes wrong rebooting is the default option to get things working again.

Though some captains are so oblivious to anything technical that even rebooting is sometimes difficult ><

 

[Blazkowicz]

eh even clicking on some stuff till it works can work sometimes. It's even the most sensible thing to do - when it's a matter of busy/unavailable network resource that will be fixed a few seconds or minutes later.
So next time you see that idiot button-mashing user, think again. He's outsmarting you.

 

[Mize]

You should probably reinstall windows. In all seriousness I worked at a company where this was the ms certified it dude's #1 solution.

 

[MfA]

It solves his problem for a while.

 

[Grall]

You should probably reinstall windows.

Also, make sure the power cable is connected properly...

 

[Albuquerque]

That should be a HIPAA violation, so report them.

I'm not using their system yet, but I'd get our company legal team involved before I went quite that far. But yes, I agree it's likely a violation. I know an external auditor would never let it fly...

I think the root of the problem is that most HR folks know nothing about IT, making it very difficult for them to hire good IT personnel. The ones who are best at it let the IT staff hire the IT staff.

Where I'm working now, we actually have a fantastic HR department in terms of hiring efficacy. They don't know IT any better than you would already guess, but at the same time they are also self-aware enough to know that. They're really amicable to sitting down and talking about the right things to see on a resume, the right kind of screening questions, and that jazz.

And they understand that we're picky

 

[homerdog]

Where I'm working now, we actually have a fantastic HR department in terms of hiring efficacy. They don't know IT any better than you would already guess, but at the same time they are also self-aware enough to know that. They're really amicable to sitting down and talking about the right things to see on a resume, the right kind of screening questions, and that jazz.

And they understand that we're picky

That is a rare thing. Treasure it :smile:

 

[Gubbi]

We have one guy that does all our phone support (not much, but anyway). He consider the people on the other end of the line equivalent to these guys

Cheers

 

[dZeus]

The organization that employs me is a bit small, perhaps 275 employees by the end of this year. We are a not-for-profit medicare provider for a small region in the state of Kentucky. My staff is a total of five -- three service desk folk, and two infrastructure folk.

Most of the employees here use a data entry system based in a hosted Citrix system provided by a much larger organization (a well-recognized name in the medical data field, but who shall remain nameless) based in another state on the eastern seaboard.

They are migrating their website to a new URL, and a month ago informed me that the "old" URL would stop functioning soon and that my users needed to change their bookmarks and links. By itself, that's a fair ask and something that my team would certainly handle.

Except, their new URL has a broken SSL certificate. When I visit the new URL, the browser warns that the certificate FQDN does not match the URL that was entered. Now, as an IT "guy", I get what is broken and I understand that it's probably just an oversight on their part. A glaring oversight that should've been caught on their very first attempt to use the site, but whatever.

I send them a note: "Hey, your cert is b0rked. I really can't migrate users to a broken SSL site until you get that fixed. Lemme know when you're done."

Nothing. For three weeks. I finally sent another note at the beginning of this week, after noticing they still haven't fixed it, saying something akin to (but not verbatim of course): "Hey, uh, so this is still just as broken today as it was three weeks ago. Any ETA on this so that we can get moving?"

The response I get, distilled into one sentence: "What browser are you using?"

Wait, what the fuck question is that? I reply back, with the obvious: "No, you dolt. This isn't a browser error, this is YOUR CERTIFICATE IS BROKEN. Go fix your shit."

They want a screen capture, in IE, because they think it's a ??? browser issue ???.

Now I'm angry. I send them a screencap in IE8, IE9, IE10, the IE11 beta, from Chrome, from FireFox, from Opera, from Mozilla, from Safari, and even took a screencap from my android phone on the AT&T network -- all of them showing the certificate failure.

Their response,distilled into yet another single sentence: "Oh, yeah just like CONTINUE and it will be fine."

This is a system that houses personally identifiable health information, which is absolutely under the strict guidelines specifically outlined in HIPAA law. Our only source of income is federal and state funding for Medicare. And your'e telling me that I should teach 200 of my users to IGNORE A CERTIFICATE FAILURE that specifically says "THIS ISN"T THE WEBSITE YOU THINK IT IS", and click Continue, and it will be fine?!?

WHAT.

THE.

FUCK.

I sent back the note simply saying "No, I will not accept that solution. You will fix your SSL certificate for your website, or my legal team will talk to your legal team about breach of contract."

It's a damned SSL cert, I just bought a wildcard cert for my company's domain name for $700 for five years. Quit being cheapasses and FIX YOUR SHIT.

</rant>

Wildcards in certificates can be very bad imo. One private key theft away from compromising your entire PKI infrastructure, depending on whether it's a wildcard for the entire domain, or only one of its subdomains.

Further more, the SSL CA model is completely borked. Every recognized CA has the ability to sign certificates for any domain. Ever checked how many CAs are recognized in your OS or browser? There are a lot... if only one of them is willing (or gets hacked) to sign a certificate for e.g. gmail.com, the person who obtains this signed certificate (and has the corresponding private key) can perform man-in-the-middle attacks against you, without you knowing.

And this has happened repeatedly already...

so yes, it's very sloppy on their part, but it's not like SSL gives you any guaranteed security in webbrowsers or other applications where you cannot limit the trusted CA for the certificate to the one of your choice.

 

[homerdog]

But it is required for regulatory reasons, and if a certificate is hacked, that is not on him.

If he instructs users to ignore certificate errors, that is on him.

 

[Albuquerque]

Homerdog has it exactly: despite weaknesses in SSL that I simply have no control over, I still cannot teach my users to ignore blatant errors such as these.

It's akin to saying that Windows is so insecure by itself that I should just go ahead and let everyone have admin rights to their own machines, and then telling them to "ignore" the UAC elevation prompts. Except none of that actually makes sense -- despite the vulnerabilities, there is a LOT of prevention that comes from simple due diligence.

This issue is now resolved anyway; they actually DID have the fix already, but their DNS was providing the wrong IP interface. A minor swap to the A-record and the website started working as intended.

 

[dZeus]

But it is required for regulatory reasons, and if a certificate is hacked, that is not on him.

If he instructs users to ignore certificate errors, that is on him.

Absolutely true. However, I found the rant against not properly configuring a certificate in the same post as talking about a wildcard certificate had some comic value.