Coping with Windows update 2661254 SSL FIx on a self certed domino server

Well well well, that was an interesting couple of days!
Microsoft in their infinite wisdom decided to release a fix that stops Internet Explorer (a pox on it) accessing any SSL site that has a <=512bit certificate.

Here is a link to the MS document

Now there are very good, sound and security related reasons for MS to do this but it did cause me some fretting and sleepless nights this week, i have no doubt once the fuss dies down you will see Firefox, Opera, Safari and the rest follow suit.

We protect a lot of our internally accessed data with self certed SSL certificates and these where created back in the days when 512 bytes was more than ample and secure for this purpose. These certs were renewed each year and over time have been forgotten about.

The symptoms of the problem post this fix being applied are:-

When IE <=8 tried to connect you get a "there is a problem with the web site" error and you can go no further, with IE >=9 you get the "There is a problem with the certificate" message but clicking on the "Proceed to the web site (not recommended)" does nothing.

Now on a i5 server (used to be the iSeries Server or AS400 and nicknamed iBoxes) renewing a self certed SSL server certificate is dead easy and you get the option to change the bit length so for our iBoxes it was dead easy. Admin Panel, renew certificate, change bit length, create, apply , restart server .. and the problem went away.

But Domino servers ... ahhhh ....

I went into the Certificate authority NSF created for the server, tried to create a new certificate, not a problem but no field to let me change the key length.. ARRRRRRGGHHHHHHH!!! Tried a whole lot of things to get a 1024 long key, with no great success. So my quickr users on the one server with the problem had to start using another browser whilst I sorted this problem out.

As it turns out Per Lausten Domino Guru and all around nice chap tweeted a link that lead me to the solution ...many thanks Per!!! Once again Social Networking helps the poor benighted admin out of a tight corner not really of his own making.

Basically what I did was the following which I was missing in the other ways I tried
The full details for what follows are on the link above, but in summary you just start from scratch.

01. I took a copy of the original Cert Authority NSF created for the server
02. I took a copy of the .key and .sth files currently in use on the server
03. I created a new nsf using the Domino Certificate Authority template CCA50.ntf
04. I created a new CERTIFICATE AUTHORITY KEY RING FILE with a 1024 bit key (option 1)
05. I ran the Configure Certificate Authority Profile (option 2) for the new key ring file
06. Set the expiry to 5 years
07. I ran option 3 - Create Server Key Ring & Certificate, filled in the guff required paying special attention to put CAKeyPair in as  the CA Certificate Label and the fully qualified domain name of the server as the "Common Name" and 1024 as the key length.

Et Voila!!! I have a new Keyfile.kyr and KeyFile.sth with 1024bit keys!

All that was left was to copy these to the server and stop and start the HTTP task and IE started to work again, which was accompanied by a massive sigh of relief and a couple of memo's suggesting we might as well go the whole hog and get "real" certificates even thought they cost money.

Thanks again Per for the link that got this sorted you are a star!

A wee problem with FIrefox and Quickr

Came across a problem with Firefox 4 and 5 today. It is a rather esoteric problem but it is a pain in the arse if you get it. I upgraded to FF4 last week and as soon as I did Quickr stopped working. When I went to the quickr home page I got 6 errors logged in Firebug starting with an "unterminated string error". this only seemed to be affecting me, the tech support chappies were all OK with both Firefox 4 and 5.

I started disabling plugins and addons and it was when I disabled the Lotuslive chat extension that the problem went away. Rather than just blame the Lotus Live extension I uninstalled FF and all the add-ons and extensions and installed a clean version 5.0.1 with no add-ons and re-added Firebug, Flashbug and the LotusLive chat extension (in that order) and Quickr continuted to work normally, so I would guess that it was a more complex issue that just the Lotuslive chat extension that caused the problem.

So if you come across Firefox not working for quickr and you get 6 errors starting with an "Unterminated Sting" error I suggest you uninstall FF and then do a clean install and everything should be fine

WARNING the Domino 852* ID Vault does no play well with 850* client code stream and there is no fix!

This is by way of a warning to anyone who uses ID Vault and has clients on the 850* code stream and is considering moving their domino servers up to the 852 code stream - DON'T DO IT!

Last week we moved one of our production servers up to Domino 852FP2 (300+ mail clients) and the Admin server for the domain started to get very very very slow, bandwidth vanished and everything all over the 20+ servers suddenly got very groggy.

We quickly tracked the problem to ADMIN4.NSF it was 2.4Gb in size and had over 6 million documents in it. 5,996,891 of them where HTTP Password Change requests originating on the server we just upgraded to 8.5.2FP2. A morning of testing showed that any Notes Client attaching to the affected server that was on the 850* code stream was generating an HTTP Password request everytime it started an NRPC session with the server. In some connection senarios 1000's of these requests were appearing a second. These then replicated from the affected server over to the ADMIN server in Ireland where is was actioned, updated the NAB and then replicated the NAB and ADMIN back to the affected server in the Czech Repulic and then out to all the other servers in the domain ... what fun!!!
A road warrior user reported a client directory replications of 26,000+ updates over the space of 12 hours.

We got around this initially by changing the ACL on ADMIN4.NSF and Denying Access to the affected server. This stopped the HTTP Requests appearing in ADMIN4 and allowed the network to return to normal. In the meantime I turned off the SYNC Internet password in the sec policy for that server pushed out the policy and then relaxed the ACL and the HTTP requests disappeared a quick agent run later ADMIN4,nsf had all the rogue records deleted and it had been compacted.
Now the change to "non-syncing the HTTP password", while being outside the strictest interpretation of our security policy was no "real" problem short term as the password change intervals gave us a couple of weeks of grace before this becomes a real problem.

I opened a PMR with IBM and informed them of the situation and the facts of the case. Their response was quick and unexpected- Ok upgrade all your clients to the 851 or 852 code stream ... that is the fix ... can we close the PMR?
WFT? 
Sorry what?
Could you repeat that?
A potential server and domain threatening bug which IBM acknowledge is a problem in code that at both 850* and upwards are within support and there is and will be no fix, just the advice to upgrade all your 850* clients to fix the problem ???

I rechecked the Upgrade Instructions and not a hint, link or suggestion that this could or would be a problem. Needless to say had there been a "If you are running... etc" warning I would NOT have upgraded the fecking server until all the clients were on the 851 code stream or better. There are Tech Notes that mention it at other releases but not the combination we had. Since there were no actual errors as such and the problem was silent when tested on a standalone server in a standalone domain where the rogue HTTP server requests were all created and actioned and resolved quickly and with no replication or heavy user load, this problem went un-noticed.


Now a quick check on the interweb showed that this has been a problem on an off for several 8* releases and it has been addressed previously ... I am asking myself how does the same bug get back into the code stream with such seeming regularity ? The answer to that I will leave you to make your own mind up about .. but I am thinking change control.

So "Upgrade all your clients or roll back the server" were the only options we were given to answer our PMR followed by a request for it to be closed. Well in the situation why the hell ask me if the PMR can be closed?  For me there is still a problem, it is a bug and it has NOT been addressed other than to provide something unexpected, unplanned and unbudgeted upgrades to 200 odd clients before the end of the month. Even so I am assured that the PMR will not result in a fix and that old chum is that!

I made sure that my reluctance to close the PMR was noted and that I thought my expectation of "support" for an active code stream was very markedly different from IBM's I also stressed that it would have been helpful if this problem had been written in a nice large font in the upgrade notes.

IF YOU ARE USING ID-VAULT AND HAVE 850 CLIENTS DO NOT UPGRADE.
would have helped more than the suggested fix.

A colleague had encouraged me to open this PMR and even though my already somewhat jaundiced view of support being offered not only by IBM but other big companies made me reluctant to do so, however the idea that "Well things will never get fixed if you dont report them" convinced me that I should be a good upstanding net-citizen and report the problem.

I did .. and it gives me no satisfaction to report my experience was a factor of annoyance many times worse that even this grumpy 30 year IT industry cynic could have imagined. Will I waste my time reporting an issue the next time we have one, or will I just not bother and find my own way to work around the problem?

Bah Humbug!

Anyway rant aside.. please do be careful if you are planning an ugrade, while this will not crash your servers it does have the effect of ADMINP jobs taking 70+% of the CPUage and your bandwidth will probably drop to 1989 speeds and remember IBM don't mention this in the upgrade notes so it will take you by surprise.

** UPDATE ** I forgot to mention the Replication Conflicts in the NAB if you have HUB and SPOKE servers.. since the NAB is being updated several times a second you get lots and lots of replication conflicts when the remote servers can't keep up

NoteMan Saves the day AGAIN!

It is nice to give a product a "call out" once in a while and today is one of those days.

I had an app today where a unintentional download FUBARed exchange rates over a large number of records. Now I could have written an agent and fixed it that way however in my essential notes tool box I have a copy of NoteMan  and I was able to go into the app, created a collection of docs that needed changed , did the change, a quick "Compute with Form" and bob is your da's brother, docs all back at sensible exchange rates.

BRILLIANT!

Well done those nice people at Martin Scott and if you need a really shit hot notes swiss army knife to help you when things get "awkward" this is the one for you and will be money very very well spent.

IIS fronting one of my web servers - Thanks to all who helped!!

When Isaac Newton quipped that he was "standing on the shoulders of giants" he was pretty close to the mark when it comes to the Lotus network of folk on the Internet. Ask a question and within the day a problem that had caused me major indigestion and irritability was gone., all because we are a community that takes the time to help each other. This isn't the first time it has happened but each time it does makes me proud to be part of a group that is so open to helping each other!

A big Thank you to all who suggested things, each suggestion took me further down the road to understanding and eventual resolution of the problem You are all GREAT I salute you!

Ooops a bit of a Admin "Gotcha" for me today

Time for a Mea Culpa which caught me out badly today.

ALWAYS Check that your [NET ADDRESS] field on the PORTS tab of the Server Record is set to be something other than an IP address.

This happens to be something that just slips my mind and if you do forget and put an IP address in there. It can reach up and bite you on the bum like it did to me today when I spent a whole morning wondering why the F**K a server was communicating to an old IP address which wasn't assigned by the DNS, wasn't in a HOST file, wasn't on the connection documents. The server just wouldn't be told.

Then I remembered the Server doc NET ADDRESS fields.. and yes there to my intense embarrassment was an dotted decimal address where a host name should be. It had been probably been there for years and it was only now when the physical hardware was swapped out that everything stopped working.

So let my blushes be a warning... don't forget your NET ADDRESS fields!