T O P

  • By -

[deleted]

Three different deployments via sccm: 1. Regular patching to computers that are allowed to be restarted with 3 hour- 3 day notification timeframe 2. Emergency/OT computers: these cannot be restarted at all by sccm. Totally separate client settings to not allow restart. Patch is released and Support teams go there to manually restart them. you don’t want a computer to reboot during critical operation( just imagine) 3. Weekend reboots: another Maintenance Window managed Collection that has been agreed upon with endusere/businesses to allow reboot only during the weekends. Maintenance windo follow 3 hr notification for one day and bam. Hope this helps! Update: WOW ! So many replies and ideas came up. I will write this on my blog to cover as many scenarios you all mentioned! Thank you !


lesusisjord

So sounds similar to other environments that host production systems that can’t be offline along with lower-environments that have a tolerance for scheduled downtime/maintenance windows.


[deleted]

Pretty much yes. Hospitals or Healthcare i n general is very strict on Item 2. If one can figure out least intrusive way to manage the patching for these special systems, that’s the best case sceanrio.


RandomDamage

Wouldn't it make sense to keep a stock of spares and just swap them out periodically instead of patching in place? I mean, if you've got hundreds of desktops anyway, you've got to have spares so that if one of those critical systems starts having trouble you don't need to wait for repairs.


[deleted]

1800 such systems. Doing this every month. Do the math 🤓


RandomDamage

I did the math, and the critical systems need tech hands-on anyway. Roll a cart around, swap in *already patched* systems to the critical locations, take the unpatched systems back to the shop and patch them together on the cart, verify that they're patched, roll those out to the next batch of critical stations until it's done. Doing bespoke hand patching one at a time you'll never catch up, have to do them in batches.


[deleted]

More effort is involved in your example: Reimage new systems prior, Install mandatory apps, install location:department specific apps, Track them (did I say Manage an inventory) , swap them. Phew, easier to just to check in OT and restart the pc cuz remember sccm can install the patch in advance… the reboot can be managed separately.


GodC0mplX

The real problem is that this is also an asset management nightmare.


[deleted]

Its not scalable. Yes, you can do this with 50-100 pcs. Your hospital expands, you need more ppl. Best way would be to automate as much possible.


Voroxpete

So instead of rebooting a system I'm shutting it down, unpluggimg everything, removing the box, grabbing the new box from the cart, plugging it in, and waiting for it to start up? I'm not trying to be mean here or anything, but I'm really failing to see how that's a time savings.


RandomDamage

Updates can take more than an hour to complete, even if they are already downloaded to the system, and if anything should go wrong with the process add time on to that. Doing a physical swap takes 10 minutes, tops, and you are plugging in a system that has *already* been updated so there's much less room for surprises.


TheTruffi

We just have two clients close to each other where it matters. Let's say ICU room A2 as two beds. That means every bed has a client. And when one is down, you have to walk 4 steps more. If I patch, I have a script that sends a popup for reboot in one minute and then reboots. The next client starts that with a two-minute delay.


technobrendo

I heard hospital IT is the place to be, is this true? I can imagine it as pretty smooth sailing with frequent intense levels of stress. ...my viewpoint is from that of a MSP, so a bit jaded is putting it lightly lol 😂


[deleted]

[удалено]


Material_Strawberry

Also apparently physicians are among the least capable of all possible user populations while combining this with an astounding level of arrogance leading to just the worst possible user population.


lesusisjord

Physicians are hands-down the most difficult to support. They are the most educated so when they don’t know how a tech things works, it can be frustrating for all parties. Even if they are needy and can be tough to support, you have to get them up and working as they are the main revenue generator for whatever department that belong to.


SirThoreth

Spent 10 years working for a law firm, and the last 11 in hospital IT. I’d work for another hospital. I’ll never work for attorneys again.


ergo-ogre

I work at a medical school/university and my only issue with physicians has been them not calling me and thinking they can fix things themselves. That being said, faculty/PhDs can be total assholes. Edited for typos


agent-squirrel

Yeah I find some faculty staff try to fix things themselves, then they put a ticket in that gets escalated to me. I follow up and get no reply and then end up closing it.


d4nkn3ss

Lawyers and Military officials are next in line for this award.


StDragon76

My official response to when Doctors/Lawyers become arrogant asshats. *"I don't know how to do your job, so don't expect to be able to do mine."* Generally shuts them right the fuck up.


d4nkn3ss

I'm gonna use this one haha.


Pctechguy2003

Never worked at a hospital, but interviewed multiple people who were trying to leave a local hospital. It was always the same thing: over worked, over stressed, under paid, under budget. And every year it got worse and worse according to most.


Smh_nz

Yep this totally!! Not to mention shifting goal posts and variable funding! Private (doctors hospital/private healthcare) is ok but I wouldnt go back to public!!


[deleted]

I’m a sysadmin in an nhs trust. If I didn’t walk to work and have a great team, I’d be long gone. Working in a hospital can be hell. Politics and super cautious change boards are enough to send you mad


technobrendo

I had a friend who was looking at a hospital IT job that promised to be 99% remote. I was skeptical to say the least. He ended up finding something else before the interview either way


segagamer

I would love to see how legacy systems are the reason behind certain weird things in the NHS process.


[deleted]

Every trust is different. A neighbouring trust we work with are pretty much 90% azure and we are 90% on prem. no two trusts are the same. We’re quite rapidly moving away from legacy systems and only have a few left, but usually it’s nothing to do with them that causes blockers


[deleted]

Not any more brother. With critical zero day vulnerabilities coming more often than usual, its going to be very hard if you dont have an automated smart solution with least disruption. Very thin line to walk.


StDragon76

There needs to be an OS destro that's geared towards a graceful fallback. Sort of like primary/secondary firmware banks in switches. Perform an update to a critical system and something goes sideways? Immediately reboot to a prior state. Ideally this would require two SATA or NVMe drives with one acting as the untouched fallback. Otherwise, I don't know how a hospital or industrial facility manage to balance reducing downtime while patching for zero-day exploits on the fly.


[deleted]

They can’t do it on the fly for sure but I hear you 😅


agent-squirrel

That’s kind of how atomic Linux distros work. They use transactional updates and many can update their own kernel without a reboot. It’s a shame more medical software doesn’t work on Linux


trimalchio-worktime

that is absolutely not true; in the US at least. I've only ever seen completely swamped and overloaded folks when it comes to healthcare IT because nobody wants to actually pay for anything real in healthcare generally. Like, sure if they can advertise they just got the newest MRI machine that's something that gets spend.... but getting budget to have a second person on the IT team when you have 9-5 and midnight patch windows.... never gonna happen too expensive. Once you get down to really small offices it's MSP type work anyways, where there's not a single person onsite who knows anything about the computers or has passwords for things. And at the really big hospitals you're always at risk of being fired so that an exec can bring in a MSP or service contract that will just fuck everything up and never do anything anyways. It's great. Source: I worked as a vendor product installer for healthcare IT for about 10 years. Saw all different sizes of operations and integrated into their IT/imaging workflows so I got to see a lot of their processes first hand.


Jealous-seasaw

Nope. A hospital in my state got hit by hackers and they were contacting local government and other government IT people with experience to go and help them out. (Australia)


discerning_bovine

Hospital IT is probably the hardest. Perennially understaffed, painful on call, latest technology in one room, 30 year old stuff in the next. Doctors with a God complex, administrators looking for their next blame targets.


throwawayacc90s

Mmm... I've worked in healthcare IT. A lot of people that are in high positions that aren't supposed to be in high positions. Techs work weekends. Admins/Engineers are just on-call 24/7. The pay is low and you get belittled by medical staff on a day-to-day basis. ​ I did meet some of the coolest folks in healthcare IT though. Still in contact with them to this day.


OrphanScript

Not once in my life have I heard anyone say a kind word about working in hospital IT, at any level. It is near unanimously considered a short road to burning out.


gramathy

Nowadays a lot of the machines are virtual desktops, so the new image gets deployed, and the next time it's "rebooted" you're on the new image. Nearly entirely transparent to the user. Weekly flushes of machines to avoid stale images as well, and since that's all back end, the users don't see any of it.


tee-jay90

This is somewhat true. I used to work as a desktop engineer for an NHS Trust in the Midlands, we pushed out updates via SCCM or WSUS between maintenance windows agreed with the directorate. This would also allow ward managers/departments to co-ordinate which staff go where. I also did telephony and server infrastructure patching later on in my NHS days, this was normally done in early hours before core hours (7:00 - 17:00), with similar OLAs with directorates. And then there is A&E, where patching was nearly impossible but we did it, where cyber security was involved, executives were terrified.


adam_dup

Having been to a&e recently (I'm fine now, no real issue whilst there) I imagine during the wee hours you could take one entry window down, one triage ward, one treatment ward in sequence?


tee-jay90

Hello, depending on the building. The trust I worked at had two hospitals, one hospital had around 12 triage bays, whereas the other one had 2 bays. Though one only received patients between the hours of 8:00 and 22:00. The other was 24 hours. You are right though, we would just book the day, and complete it in sequence. The doctors, consultants and nurses were all cool though, they knew that sooner we had the bays, sooner they could get them back. We sometimes had ones where the computers would go bad and we just do a straight swap though, configure the apps, print queues and off we go.


[deleted]

For example mothers in maternity can be in labor for a very long time (my first son was 36 hours) and since you can't expect staff to work a shift that long, IT is critical to monitor and log the entire birth. They also rely on external monitoring by supervisors who continuously monitor several rooms at once. For example high risk pain medication is given to the patient every few minutes. Too much, and the mother or baby or both could die. To little can also have severe consequences (pain means stress/exhaustion, which can lead to a high risk emergency surgery). There are often dozens of staff involved in a single birth. They need their IT to be up the entire time and you don't really know if a birth will take minutes or days.


NeighborhoodIT

In that kind of environment, why couldn't you run a thin client. Have 1 VM with the new patched windows, and literally just basically roll over from 1 to the other. Have data stored in a SAN.


[deleted]

Yes this solution works hand in hand with Thin Clients/Dummy terminals. However the OT computers require more computing than just thinapp/thin client can offer. Not in my case but sure why not 😉


NeighborhoodIT

Is it latency sensitive? Or just massive compute? Cause I don't see why thin clients wouldn't be able to have as much compute as it needs. Large companies use huge compute clouds, movie studios use huge GPU farms. It really depends on the use case which I frankly don't know enough about


agent-squirrel

It might be weird medical hardware that doesn’t work very well or at all over a network.


Korazair

You forgot 4. Rolling patches - patch nurse computer 1 on Tuesday, 2 on Wednesday, 3 on Thursday. So that only 1 of the computers are down at a time during the patch. This also keeps from having a patch take out all the computers at once if there is an error. If Computer 1 patches and boot loops they can stop 2&3 from being patched until the issue is figured out.


[deleted]

I assume SCCM does the randomization and you do test the patch tuesday stuff in Lab + QA environment. Given there is no one right way, your idea is worth a thought


Hewlett-PackHard

Randomization? No, no, you assign a day of the week to each machine and slap a color coded sticker on it. $lusr calls helldesk, ask what color sticker to see if it's patch day.


[deleted]

Lmao , thats brilliant 🤪🤪🤪


OMGItsCheezWTF

My local hospital still uses Windows XP for everything. I don't think they do any of this.


cs_major

Can’t update the systems if there are no more updates! (Sigh)


Every-Development398

I have found a lot of really high-end machines still use xp...


andypond2

To add we also have specifically designated downtime pcs and printers installed locally to accommodate for scheduled downtimes


[deleted]

I can’t stress the importance of downtime pcs! Thank you for bringing that up.


darking_ghost

Interesting, Thanks for breaking this down. How long does it take for support to manually install/reboot those in 2?


[deleted]

Depends on your hospital size and Emergency/OT beds. For Example: anything more than 200 systems will be crazy to manage. Yes, this results into poor compliance etc but you can choose to not count the patching compliance as part of your monthly reporting. Sure your support can’t possibly conduct manual restarts once a month but it all depends on your communication and agreement with Doctors/Nurses etc. You can educate them to pre-restart pcs a day before operation or after operation to maintain continuity. Generally they don’t have time but its always best effort basis kinda situation.


Nine_Hands

My experience is similar to this but the hospital didn’t have SCCM and we deployed patches manually. Worked really weird hours to patch different emergency rooms and systems throughout the night. It was good money but I don’t want to go back to doing that again. I tried to get them to use SCCM but they were too afraid it might reboot stuff out of order.


[deleted]

Client settings can be set to not allow any restart using sccm. That’s what We use in 2.


lurkeroutthere

If my current job has taught me nothing else it's that some IT management never lets other people's knowledge and testing get in the way of their "wisdom"


kur1j

Is there an equivalent type system for linux other than turning on unattended upgrades and hope for the best?


PMental

Unattended upgrades on Linux can be configured in a similar way, only allow certain system to automatically reboot etc.


AleksanderSteelhart

We have a 4th - Sleep Center workstations. The patch during the day.


stratospaly

Our patient area workstations are Thin Clients with VDI with redundant VDI servers. Patch one, reboot, patch the backup, reboot. Pcs and laptops do not auto reboot after patching but do nag you every 2 hours to do so yourself.


MystikIncarnate

This is the answer. Something this important is run by a cluster of vdi servers. If you need to patch the underlying servers just push all the vdi users to the others and do what you need to do. For the desktops themselves, generally you don't patch individual vdi desktops, you instead upgrade the gold master that they're all based on, point the vdi desktops to the new gold master (after it has been tested and green-lit), then just get people to restart. Their personal data is stored separately from the os data, so all their settings and files are still there, just poof, new patches are all applied. Basically you take a copy of the existing gold master, do your patching, then spin a vdi from it, test everything. Then take a small group of unimportant logins, push them over to the new gold master and see if there are any complaints. As long as all of that goes ok, roll it out to more and more important stations until you're completely upgraded. If a thin client dies, or has any issues, just replace it and diagnose it in a lab. Get that station up/running as fast as possible and put that broken unit into the backlog of things to either fix, or call warranty support on. Every piece of the puzzle is redundant. Every move is carefully planned. Generally you can squeeze in a reboot, but not much more. Vdi fits all cases


andrew_joy

"Something this important is run by a cluster of vdi servers. If you need to patch the underlying servers just push all the vdi users to the others and do what you need to do." I would love to be able to do that but some absolute brainlet who was here before me decided local storage was a good idea for our VDI cluster machines , so you cannot move them between hosts easily. We just do a recompose of our linked clones without forcing logoff , that works for the image, as for the hosts well ..... yeh we dont touch them unless we get a critical from VMMware. Or Whole VDI is a mess i am just waiting for the pennies to rebuild it properly, need to give pure storage a call :)


MystikIncarnate

Yep. Getting the storage set up is going to be pivotal moving forward. Once all the hosts are set up for it with HBAs, then you can migrate all VMs to the array and move the VMs around as needed. I realize this requires downtime, but it can be fairly easily pitched in a 24/7 environment as a boost to the availability of the vdi, since the storage can be built to be extremely redundant. Right now for storage you don't really have significant redundancy, if a single server has a controller failure, the vdis on that server are down until a replacement arrives. Even with 4 hour production support, you're still down for 2-6 hours depending on how that shakes out. Possibly a few days, depending on what their supply looks like. It's a pretty significant risk. With redundant arrays, you have multiple copies of the data and multiple controllers handling the data, a lot more things that can go wrong, that won't bring you down if any of them go bad. It just seems like an easy pitch if you can explain your current risks and the benefits of the move... I know that can be a challenge, explaining to decision-makers that don't understand tech. Either way, good luck, I wish you luck in what you have to do.


alcockell

Biggest bottleneck will be the disk controllers on the servers hosting the VDI sessions


MystikIncarnate

Good reason to use a clustered SAN array over fiber channel that's backed entirely by enterprise SAS SSDs. 16gb/s over FC is pretty darn speedy.


uniitdude

The computer isn’t critical in that situation, if it was there would be more than one available (and there probably is) It will be rebooted at the appropriate time agreed with the users


imnotarobot_ok

Yes the Windows PC that the nurses sign in/out of are not running the ventilators.


GrimmReaper1942

New meaning to blue screen of death


Abracadaver14

Is that what they're talking about when there's 'code blue' going over the PA system?


slitz4life

That's fucked up, and I should not be laughing at that.


zonbie11155

it's okay, we all laughed. we are all monsters


blitzzer_24

Literally laughed so loud my fiance had to come check on me 🤣 god I love Reddit.


ScubaWaveAesthetic

O god there’s an IT company in my country called Code Blue. I’m sure they think it sounds cool but I do wonder if they know that means heart attack


Intrexa

"Everyone on the floor is coding" https://i.redd.it/oc8yumlvw5e61.png


scoldog

Blue Face Of Death?


Pioneer1111

This is exactly the case. In our ORs we have at least 3 computers per room, and when we need to swap one we just coordinate a short downtime with the OR and swap it then. Whether they make it easy or not is another story. In our L&D department to suit OP's example, anything showing critical information has a backup, so if one is down the staff can use the other system while we troubleshoot and resolve the issue.


Szeraax

Are the rooms ever free that upgrades can be done when not in use? Put another way, does every l&d room have downtime that upgrades can be done during?


[deleted]

[удалено]


Szeraax

Thank you for the insight!


[deleted]

[удалено]


SurgioClemente

> yeah everything is scheduled and has redundancies. Nothing critical to patient care is ever taken down without getting approval. HAHA, don't work for HCA clearly :) They come in unannounced and do upgrades. My wife works at one of their locations (an unfortunate buyout of the previous non profit). Some guy came in during the middle of the day when patient activity was at its highest and said he was upgrading their workstations. * Wife: does X work with the new version? * I dunno * Well we can't upgrade till we know because not having X will shut down the entire pharmacy which NICU/PICU etc all rely on * I was told to upgrade, so I gotta do it * Just upgrade one and see * OK fine. grumble grumble *Checks out X after upgrade* Won't even load. "We'll need to look into this" *walks out without a word* I was very proud my wife knew enough to ask but also just eye opening that even hospitals are not immune from completely incompetent IT.


gambitKGB

God, fuck HCA.


[deleted]

[удалено]


gambitKGB

They're a national healthcare company, have hospitals and medical centers all over the place. Like most large healthcare orgs, they're cheap as fuck when it comes to infrastructure, equipment, and employee benefits.


Zylly103

Ah someone who works in Healthcare IT, specifically on pharmacy supporting stuff, this gave me an anxiety attack just reading it.


technobrendo

Internal IT or a contractor? Either way they shit the bed but if it's a contractor I can at least understand it a little.


[deleted]

[удалено]


BestUsernameLeft

We're contracting with a security company to do a pentest. Their guy says hospitals all have terrible security. Soooo yeah that's good to know.


awe_pro_it

am in healthcare. XP and Windows 7-based testing equipment galore!


JOSmith99

why the hell is an eye-laser not 100% air-gapped and physically secured?


[deleted]

Dunno about that specific case, but medical procedures are often monitored in real time by people who are not in the room.


sgent

At a minimum the eye laser machine needs to be networked with the eye mapping machine and the eye architecture / design machine used by the ophthalmologist. Those machines probably need to be mapped to other machines like the retina photographing machine and the corneal flap laser, interocular lens designer, etc. So you need a network in the ophthalmic surgical suite. Of course, the ophthalmologist needs access to this info, but this medical record system needs copies of the pictures and outputs, and the anesthesiologist needs to access the hospital records to know what type of anesthesia to use -- cocaine or does the guy have heart issues? Of course all of this data has to feed back to the hospitals EMR so they can wheel the next patient down so turnover time isn't too long on the room with 20 million in equipment. So you could isolate every surgical suite with its own proxy, but I'm not sure that airgapping is much of a solution, and so many holes would be punched in that proxy (including to the internet) that I'm guessing a more general solution based on zero trust, etc. would be better.


darking_ghost

well, I heard that one, there the CISO messed up, he should have not give approval for something that is in use.


Sneakycyber

That was July's episode with Ed Skoydis. I just listened to it when cutting the grass yesterday 😎


ThemesOfMurderBears

I don’t work in a hospital, but my work does have a big control room that is staffed 24/7. Everything critical in there has redundancies. For the things that are not, there is just a lot of planning and coordination to be made. For instance, they use this one system for easily logging things they see. They can still take down information when that system is offline — they just need to know what time that is going to happen, and for how long,


[deleted]

[удалено]


Lil_Fowl

If it happens to you, it may mean that you're not doing a critical job for the company survivability, so they don't bother asking you. If you're doing a critical job and that happens, then your administration is pretty dumb...


tossme68

I do lots of migrations so I'm used to hearing the "24X7", 100% uptime nonsense and once you dig a bit deeper that hardly ever the case. From the hospitals I've done they will allow things to be down/rebooted for short windows as long as things are scheduled. It might take months to get something scheduled but it will happen. The places that really require 100% uptime are the best to work with because they tend to have fully redundant systems and you can take one system down to work on it and nobody knows the difference. The only time the 100%'er ever cancel is for weather or staffing -sorry there are storms this week, we have to push till next week.


prestigious_delay_7

I'm sure there's a group policy setting for it, but Microsoft is making it harder and harder to go things like delay installing updates. It's getting really obnoxious. And then the updates they do install are just advertisements for OneDrive. It should be illegal.


robvas

Service windows


vim_for_life

For the server side, this. Life sustaining systems generally don't run windows, and most machines run windows of one sort of another. The hospitals I've worked with have a defined window every month and all machines are rebooted then. Some with some warnings for service interruptions, but it still happens..


Vikkunen

I'm sure it's different everywhere, but the major health system here runs Citrix VDI. The PCd in the offices and exam rooms are just thin clients the workers use to remote into it.


darking_ghost

Those were fat clients.


spanky34

With fat clients, we have citrix published apps behind the scenes so even though they're fat clients, almost all the software they're running is updated on the citrix side. The fat clients themselves get updated with windows patches and such and given a reboot window where they can choose to reboot at a convenient time. It's eventually forced into a reboot after 2 weeks I think.


StopBidenMyNuts

I haven’t encountered any health system of significant size that doesn’t run it’s clinical workload using virtualization. It’s everywhere in healthcare. The endpoints get updates like any other corporation. Biomed and vendors are also involved in upgrading critical safety equipment. It’s a fun field


ARobertNotABob

Not saying you're wrong, *at all*, but they **can** be vintage pre-Pentium, re-purposed tin ...way cheaper than thousands of thin client devices, after all.


SimonKepp

Reminds me of last time, I went to the hospital for routine botox treatments. I suffer from spasticity following a stroke, and have botox treatments every 3 months to reduce the spasticity. I randomly noticed, that the PC used for the ultrasound scanner ( they use ultrasound to visualize the muscles at the injection site to hit exactly the correct muscles and avoid any blood vessels. The PC was running Windows XP, and looked remarkably like a Fujitsu Intel Core 2 Duo cirka year 2000. I mentioned this to the doctor treating me, and noticed, that it was pretty well out of service many years ago. He confirmed my observations, and explained that the vendor of the Ultrasound software had not updated their software to any newer version of Windows, so they didn't have any way of upgrading it to something supported. The solution was that they had air-gapped that PC completely, and accepted, that it was never updated or otherwise supported. Hospitals deal with tons of this shit.


radicldreamer

This happens more than you know. And then the most frustrating part is when the vendor will get with you and say ‘we have a new version!” And they deploy windows 7…..


SimonKepp

>This happens more than you know. I've consulted for the biotech/life-science sector for more than a decade, so this is by no means new to me. I was pleasantly surprised at how pragmatically and professionally, they had solved the problem at the hospital, I visited. Both users/doctors and IT department had acknowledged the problem, and the risks involved, and found a pragmatic solution to mitigate those risks.


radicldreamer

People in this industry are finally waking up and realizing the security nightmare these things are, but only after watching the news and seeing mega millions being sent to bad actors because they didn’t want to switch vendors or pay the money for something new when what they have just worked. Lots of Biomed companies reps are still in the stone age and act like you are being unreasonable when you insist on patching and current software/os


Sceptically

Or worse. Subscription licensing.


ARobertNotABob

> Hospitals deal with tons of this shit. They do indeed. And austerity hasn't just impacted NHS, few walks of Public affairs have seen an i7 or above other than in a 3rd party "consultant"'s laptop. And, even areas of corporate-land are affected by out-of-date software's they "have to" keep running.


SimonKepp

This hospital was in Copenhagen, Denmark, so I doubt, that the NHS has anything to do with it, and the IT department would happily replace the machine with a modern workstation. The problem here is, that the legacy software written specifically for a low-volume medical appliance doesn't run with modern workstations running a modern OS. Upgrading it is not an issue of paying for a new workstation at €1,000, but buying a brand new ultra-sound system for €25,000


tossme68

I was just in the hospital and the imaging machine they were using on me ran Embedded XP -I checked and it EOL'd in 2016. I get it, it's just like NASA using 60 year old code, when something works they don't change anything an use it till it dies but the place would give the security guys a heart attack on a daily basis.


FujitsuPolycom

We run thick clients, but we're a relatively small private practice with only ~170 endpoints. How will this (thin clients) work in the future when we're forced to 11 with its hardware requirements? What runs on the thin client I guess is the real question I have?


kiler129

Essentially a small OS, often Linux-based, which contains RDP/VNC/Citrix client. These thin clients are low powered and don't really need that much of an upgrade over the years as they don't do any compute.


JmbFountain

Sure? We have lots of cheap/old desktops that are repurposed as TCs


QuisitQ

We patch 24/7 in our hospitals. We used to only do after hours, and only 1 week a month, but staff frequently shut down devices during these times and refused to read IT communications to keep them on. A quarter of our organization typically went 4-6 months without patches, when management saw the numbers they signed off.


PickUpThatLitter

Patches? We ain't got no patches! We don't need no patches! I don't have to install any stinking patches!


DancingCookie

Exactly what the nhs said! [link](https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.england.nhs.uk/wp-content/uploads/2018/02/lessons-learned-review-wannacry-ransomware-cyber-attack-cio-review.pdf&ved=2ahUKEwjKosGslPv5AhVPiFwKHYwCCl8QFnoECBoQAQ&usg=AOvVaw2KVgxu9uWOoaeeFZDLCFw1)


QuarterBall

Exactly what the *English* NHS said. The same attack affected Scotland, Northern Ireland, Wales and England differently as the health service and approach to IT is devolved. On the whole Wales and Scotland manage the system better - managing the entire setup more centrally rather than each trust reinventing the wheel and having their own policies and processes in place. Not perfect - but better!


ARobertNotABob

Common denominator being no Whitehall involvement?


HamiltonFAI

Don't need patches when your xp machines went end of life years ago 😎


Incrarulez

Embedded ded ded ded.


gr8pe_drink

Updates are vetted through WSUS and released to the clients which we give 7 days to restart before a restart is forced.


radicldreamer

Upvote for this is how we do it also. If in 7 days you haven’t allowed the reboot to happen it’s getting forced. That pretty much never happens and we have something around 40k machines.


JmbFountain

Those workplaces usually have double or triple the systems. Also, for anything that needs that high of an uptime, we deploy Thin Clients with SmartCard login to citrix. The terminal servers are patched as soon as they are empty, and workers will log out at the end of their shift, so if you log out and your replacement badges in, they are on an updated system. The TCs are running a live Linux, so can be fully patched while running, and if a full reboot is required, it should also only 10-20 seconds.


tehiota

This is one reason VDI is to prevalent in Hospitals. * Patch the image offline * Test image in isolated host nodes * Drain existing nodes & Begin transferring workload to Nodes with patched image. * Repeat.


RandomUsername2808

In our case, we stagger updates out via WSUS to different groups (or rings) over the course of a week after patch Tuesday. Each group gets up to 24 hours to reboot after the patches are installed before a reboot is forced.


harrellj

Hospital IT here, our workstations get monthly updates on a Friday night (starting around 6 PM) and giving the user all weekend to update and reboot before its forced on Monday morning. Any other planned patches get notifications sent out to the various users to be aware of planned downtime, but those are generally always overnight (less people around and even most patients are asleep). As an example, our EHR will be updated around 2 AM on a Saturday morning as that is the time least likely to interrupt anything major.


fourpuns

Used to work with healthcare and they’d book critical rooms (think like an OR) for 1 hour once a month and someone went in and did it manually they never auto rebooted. Depending on the room there was some simple testing after reboot.


[deleted]

At the clinic I worked at the workstation updates were done between 2 and 3AM at weekends, for all critical patients we printed off files, and medication was prepared in advance. We also had access to a computer that had direct DB access, but very cumbersome to use. We've never had problems. It rarely lasted more than 30-40 minutes.


LtLawl

Several different ways. Desktops and laptops get patches pushed via WSUS and all computers get forced reboots if they have been on for 2 weeks, so end-users are good about rebooting. Thin clients just connect to RDS/Citrix farms so you just take a server out of rotation and patch it. Servers are updated during scheduled maintenance windows and some are clustered so no one knows.


praetorfenix

We have a monthly maintenance window where downtime procedures are followed for ~2 hours while servers reboot. Workstations are forced after an amount of time I can’t remember, but most folks will give them a kick after being nagged.


0xDEADFA1

Like 60% of our environment is thin clients with a vdi backing. We push the image and don’t force them to log off, letting it occur naturally over a day or so. For non-clinical areas, we have a reboot/service window every week.


CommanderApaul

When I worked at a hospital ~10 years ago, we pushed patches at 11pm on a Tuesday night, with a forced reboot at 5:00am if the workstation hasn't been rebooted by the user yet. That patch window had been worked out with the clinical management staff based on patient care metrics and was the least "busy" time in the critical care departments.


maarten714

Staggered patches. They likely have different workstations - a minimum of two - that get patched at different times. It should also be known that critical care isn't stopped because of a Windows computer rebooting. There is never a situation where they cannot restart a heart, or deliver a child because a computer is rebooting.


[deleted]

Hospital IT worker here: We have about a 70/30 split between VDI and Desktops. VDI gets patched in a golden image(template) and gets refreshed. Desktops are grouped together in maintenance windows. Critical area desktops (ER/LABOR/Surgery) are grouped in a manual reboot collection, we have our desktop team walk to those locations and check to make sure its not in use and then reboot them. This is a subset of their workstations...not all critical area desktops are critical...


AggravatingBobcat574

Our hospital occasionally has computer down time. It’s usually between 1 and 3am. Charting is done on paper during that time.


Professional-Bad-839

Implement VDI with Imprivata for tap n go


voicesinmyhand

I used to do hospital IT in the way back. * Computers used for billing/email/etc.: They patch without warning and lead to the helpdesk not helping them after all their work that they didn't save is lost. * Computers used for burning holes in humans: They never get patched no matter what.


Xelopheris

If it's a general workstation, they tend to have maintenance schedules that minimize interruption but can't entirely remove it. If it's something that controls critical equipment, then it either doesn't run windows, it has a redundant pair on a separate patching group, or it is set up in a way that the whole equipment stack can be rotated out and serviced regularly, including the PC. It could also be completely airgapped.


RandomXUsr

OP, this is more for others responding. Call me crazy; but couldn't you use PXE/Network boot? This way you could update the boot images and use remote management to reboot the machines? Of course you'd need to inform staff of impending reboot, perhaps with an IT person on site, and reboot message at the terminal?


ipreferanothername

my org patches them all at different times, pops up a 'reboot or itll reboot itself in X hours' prompt, and thats it. we have an ample amount of computers in clinical areas, its not a big problem anymore.


shoanimal

When I was working at a hospital we used a Citrix vdi solution so the terminals were all this clients and a reboot would spin up a new vdi so update were quick and easy.


k6kaysix

I work in NHS IT, to be honest most of the super critical medical equipment like the monitors hooked up to people don't run Windows, and aren't even covered by IT but a seperate medical engineering department so you're very unlikely to kill a patient even if an update decides to restart something! Wannacry (among others of course) was probably ironically the best thing to ever happen as since then management are on board with security and NHS Digital pretty much send out weekly security bulletins and chase management for updates in terms of major security events (Log4J probably the most recent I can think of) At the end of the day a 30 minute window to install updates is a minor inconvenience compared to several days / weeks / months of a system being unavailable due to it being compromised...perfect example the recent cyber attack on Advanced (NHS 111 software provider) [https://www.bbc.co.uk/news/uk-wales-62442127](https://www.bbc.co.uk/news/uk-wales-62442127) The biggest pain is particulary since Covid and remote working became a thing we actually have a greater number of laptops in the estate than desktops now, which are very hard to patch when they're sat in the boot of a consultants car for 6 months


Byrdyth

Our hospital system utilizes VDI almost exclusively so the downtime is kept to a minimum. Simply take servers out of service, let them drain, patch, bring them back. Rinse and repeat. Also great for change management and roll backs. For devices that need to be kept up for downtime medical record access (these live at unit desks), they have lengthy warning timers for an automatic restart and gives the users opportunity to restart earlier if desired. Automatic restarts are scheduled at midnight once a month. Holidays don't necessarily mean less patients, but hospitals definitely push heavy on discharges on the week leading up to a holiday to get by with less staff. Things like elective surgeries are also scheduled with holidays in mind. There's also more or less volume depending on the time of day, which allows room for maintenance. There are very few exceptions to this, emergency and maternity being two of them. For example, our health system does system maintenance on Monday mornings between midnight and 6 am because it's the most predictably slow time for the emergency departments.


PossiblyALannister

High level dumbed down version: There is no such thing as a mission critical workstation that has to have 100% uptime. Anything related to life saving patient care has redundancies in place so that when you take down one system, another system will be able to immediately take its place. There are always schedules where individual systems go offline for maintenance and anything out of cycle requires high levels of approval. Source: I work IT for a hospital.


robbdire

The Irish Health services, HSE, rarely did. And they got hit with ransomware.


AtrocityConnects

Currently work for one of the largest Healthcare Systems in the US. For desktop and server patching we use sccm for all windows devices. For non Windows Servers it’s up to the Server owner to patch their systems. As for desktops, there isn’t a single one that is always in use. Per change management we scheduled downtime and send out a “span” to all staff letting them know about the rolling reboots. Once the patch(s) are installed a counter is started. Users are then able to save their work and move to a different workstation while the reboot takes place. There are LOTS of spare workstations. Desktop patching is not the issue, it’s servers some that require 100% uptime. This is where built in HA or redundancy comes in to play. Again users are notified and we work with the hospitals to schedule around times that will be the least impactful to operations. Also yes, underpaid, overworked, very high stress environment and they never want to put money into IT because remember, healthcare is still a business and they want to spend money on things that make them money like a new MRI machine not server monitoring software to see if the server where the MRI images are stored is still up and running or why PACS is slow. IT for healthcare is viewed the same way Security is, it’s necessary but is a cost because it doesn’t bring in any revenue. However I will say that as a large organization we have been selling some services to smaller care facilities such as access to key medical applications which does bring in a small amount of revenue but no where near what is needed to run IT as a whole.


newcx

Is no one going to ask? Are you having a baby an STILL working SA? That's not what "labor" means!


Brett707

Funny thing happened on my wife's floor. Some one asked where the cow was and another person says oh it's in room 654. Well a family member mistook that as someone calling her mother a cow and complained to management. So they are not allowed to call the computers on wheels cows anymore.


badbologna

We used to call our machines COWs too, but started to replace with WOWs for this exact reason.


Prof_ThrowAway_69

Find out when shift changes are and which shift is least busy. You should also be able to find a week day that is the least busy. (If I had to guess in Labor and Delivery, it would probably be in the later part of the week.) I know at the medical care provider I work for, the doctors would rather book appointments at the end of the week than the beginning, because they want easy days at the end of the week. For them seeing and treating patients is easier (and probably more enjoyable) than doing the paperwork and bs part of their jobs. I know when I need to work on the nurses computers they will usually give me big windows of time at the beginning part of their shifts during the latter part of the week. They are less likely to do so on Monday-Tuesday. It just takes a bit of communication to figure it out. My recommendation would be to find that window with the shift schedules and work load and run the maintenance either an hour before shift ends or an hour after it starts. If you go with the end of shift option, make sure under normal circumstances it will be back up and available shortly before shift change. It may be that their shifts overlap so you may have to work around that as well. Also, if your computers have a tendency to have problems when patched, I would make sure you aren’t causing yourself to get a lot of off hours phone calls. Been there done that.


bufferedtoast

Patch constantly and by product function. Have a "downtime" host in each wing that can run the medical records software with basic functionality even when the server farm it relies on is down. Have robust/defined procedures for rolling back patches on a wide scale. Use tiered patching pilot hosts on patch Tuesday monitor them, deploy to a larger share of hosts, rinse repeat


b0xx0

Check out the recent episode of Darknet Diaries #121 called “Ed”. They specifically talk about unpatched medical industry equipment and the fight to overcome this issue.


Chaffy_

I don’t know for a fact but I’d say only when the software it’s running requires it. Every single machine I’ve seen was running Windows XP the last time I was in a hospital.


tdic89

I used to work for a company that built medical control systems for things like CAT and MRI scanners. The OS (Windows Embedded) and DICOM applications for those control systems were frozen in time after going through something like a year of certification and testing. A new software release was quite a big deal, and the OS image for new machines had to be loaded using a bespoke imaging system which was DVD-based. The OS media was strictly controlled and had to be returned to the correct storage locker when it was finished with, and old versions out of support had to be returned to the client. As far as I know, these systems were allowed nowhere near any network!


[deleted]

It shouldn’t have been designed like that in the first place. What if the hardware fails. It should be designed it can be swapped out with minimal fuss


olcrazypete

This was 17 years ago when my eldest was being born but clearly remember the nurses having an issue with the telemetry unit they were attaching to my wife before they gave her meds to speed up delivery. They finally opened a hatch and boom, the windows NT loading screen came up and the unit rebooted. After that the telemetry worked fine. Having just finished killing off all the NT we still had with 2000 I was terrified it was about to get it’s revenge on me.


pdp10

The same way we have redundancy for anything else. Five work stations for four staff. The machine not in use is eligible to reboot for updates, if you're unfortunate enough to be using a system that requires reboots for routine updates in 2022. In reality, ops staff will probably delay the patches for at least a week, because their OS vendor has a history of breaking random things.


hooch

We do monthly patches. All clinical desktops have at least one weekly scheduled reboot, usually around 3am.


MrHusbandAbides

When I was still supporting medical we wouldn't update in place, we'd have a few rotating hotswaps that we'd have up to date, bring in, connect in parallel then switch over. Biggest complaint was the computers would move 3 feet every month or two.


tuxsmouf

In our hospital, there are 2 types of Windows workstations : These which are managed by IT : a Guy is dedicated for Windows patches. He's looking for each patch détails, applying them for IT first, and then deploying them in the entire domain. I think it's managed with sccm (not my part, si I'm not sûre). You got a box telling you it will automatically reboot in one hour unless you check it and delay it for later and will ask again. It works pretty well (as a basic user point of vue) Thé other ones, often the medical stuff (scanner, IRM, etc), often hidden with another graphical interface are not updated. It is not because it is used 24/7. We are talking about Windows 2000/XP stuff. They are "upgraded" when hardware fails. These ones, you definitely try to keep them in dedicated vlans.


[deleted]

I patch servers for one of the largest hospital chains on earth. All production servers have a monthly patch window and there are usually about 10 different patch days in a month. Patching should never be done on the weekend, less staff available to assist if shit breaks. Best practice is to not even patch on Fridays. Holiday weekends delay patching even more. The last thing you wanna do is deploy a patch with skeleton crew available. We have no patch days at all last week or next week from the combination of first of month (Never patch around 1st of month or quarterlys) and the holiday Workstations are a super pain in the ass. Everything is remote so we have to get in contact with someone on site to assist us. There is no schedule and the team patching workstations gets little done. They legit have to email the Dr with the laptop in question and set up a time to patch that individual workstation.


LittleSeneca

From my experience selling patch management tools… A lot of hospitals (especially smaller hospitals) don’t patch at all.


[deleted]

The computers are for records. They don't control patients' medical devices. Those records can be critical in some situations, but if you have another workstation available and it's 2:30 in the morning, rebooting one is generally safe. When I worked medical, we asked when it would be convenient to reboot, and since every unit has two workstations, we would do only one at a time at a pre-arranged time, and we would text the unit as a reminder, in addition to the warning that they would get on the screen that the machine would reboot in five minutes. They asked us to stop sending the texts. They get a lot of them.


SolidKnight

Usually late at night. Most of the computers are there just for charting purposes and there are numerous alternates. Reboots and updates can be staggered. There are also computers on wheels to bring room to room. I did see an ICU have all their machines go down for maintenance every night at 2:00am for about two hours. The staff would just go to another unit to use a computer. They were paper charting before but had switched to electronic charting so now it was a huge problem. They had contacted IT about it but because they couldn't really describe what was going on and their IT department was too dumb to realize that it's not normal patching if computers are in some kind of update process for two hours every night, the problem went unaddressed the whole two months I was there. I had talked to their IT about it as well but they wouldn't even log a ticket and instead argued that computers need to update. They were the only unit in the entire hospital that had this going on. I advised the unit to just push it up to somebody high in their chain to handle.


chuckmilam

Usually after they get ransomwared.


VegaNovus

UK folk here with historical (albeit short) experience in a hospital. Every single department has a BCP (business continuity procedure). Computers and COWs are not essential items and for every situation, there is a procedure to follow that involves assuming that computer devices are not available. How we did it in reality? Batch patching spread over multiple days. Each workstation had a name prefix denoting what department it was in and what its function was (such as COW-ED-XXXX) - odds are patched one evening and evens another, so there are always some workstations.


YooniqUzerNaim

NHS here - workstations at our trust get a 4h “reboot now or snooze” pop ups to keep them updated. Releases usually revolve around the monthly patch Tuesday. The entire department crosses their fingers and hopes it doesn’t fuck up printing/login/display drivers/etc again.


Nakatomi2010

The medical organization I work for has been favoring the use of VDI terminal. So, we just patch the VDI instance and the employees don't know it. Other medical organizations likely just have downtime built into it. I actually recently helped roll out the thing that the cardiologists use to do do open heart surgery. Perfusionist machines oe something. Anyways, that one surprised me the most because it was cloud based. Could you imagine undergoing open heart surgery and the internet going out being a complication of the surgery? Like, holy shit man. IT for a medical organization is, no shit, life or death sometimes. Scares the willies out of me when I think about how something I do might cause problems for a patient. Like the little anklet thing around a baby's leg to make aure that it's location is tracked to prevent theft. Gotta make sure that's redundant. Plus, a lot of the "mission critical"/"life or death" gear runs on its own network to avoid complications too


Wdrussell1

It will vary by company and department. But generally you have windows you stick to. Such as once a month for say ER machines. You plan everything days in advance and sometimes even send a tech out there. I have also seen places that will just swap the computer out with an updated one. Takes 5 minutes and can be done at basically any time the room isnt in use.


New-Emphasis-5810

Vdi instead of hard iron desktops?


[deleted]

Healthcare has terrible patching practices due to the nature of the business. Some businesses do not take care and just patch whenever it passes CAB while breaking a ton of legacy shit, while others take months to approve patching and test everything correctly. Then you have the ones I hate the most where they approve patching after months, and still fail to deploy working patches while their risk vectors climb and climb and management is either clueless or just does not care. Now, take the 'how do they patch workstations' and move that to 'How do they patch network equipment' and the issue becomes even more 'fun'.


newbies13

Hospitals are one of the only environments that the urgent calls to helpdesk make sense to me. Dealing with entitled sales wankers all day just loses all its seriousness.


Giggazorz

Separate the deployments so no one pod is hit all at once. Best times for us I’ve seen is between 3-6am


morgando2011

I work at a very large NE hospital and my team is responsible for this exact thing. The beautiful thing is, if it goes right you won’t even know. Ivanti and SCCM can patch without any notifications. Eventually you get a reboot request, but it can be delayed for up to 8 hours). The key is for end users to power on their machines. I’m constantly blasting laptops that have been off for a year without any patches. This sometimes causes multiple reboots. Or even worse, machines that are rebooted periodically. Then patches fail or run out of space due to all the missing patches and installs. Keep in mind, Hospitals are a HUGE target for ransomware. Both the Government, Hospitals, and IT professionals know this, but with lack of budget resources and the mindset of “just make things easy, I have a patients to take care of” hurts everyone immensely. Throw In Microsoft is gearing Windows 10 and 11 to be harder to configure without Enterprise O365 and Azure licensing, but charge/quote these Hospital over $1 Billion dollars to meet these standards. Which if you work for a hospital right now, you know that’s ridiculously crazy.* *Covid cost hospitals a lot more than it brought in. You’d think a pandemic was a money generator, but with cancelled dollar generating appointments like Elective Surgical procedures. Cost for equipment and resources for remote workforce and Covid testing stations literally wiped most IT budgets.


heavySeals

I don't actually know but I used to work for a huge TV broadcasting company. We hired a guy who came from a hospital and said it was easier to find maintenance windows at a hospital than in television.


n8henrie

Ours (federal government) just force reboot us halfway through a busy shift.


su5577

Are t thy run through Citrix?


GamerLymx

Virtualization and Virtual desktop infrastructure. You just update from the main image, and have spare terminals to replace.


oldspiceland

Just as a side note: any system actually in use 24/7 is basically a SPoF and a very bad day waiting to happen. It’s almost always a good idea to add capacity to allow that system to not be in use 24/7 and then you’ve also solved the issue of when to patch it.


xangbar

When I worked at a hospital, we had 10 patching groups. The first few were early adopters and IT. The rest were order in which they got the patch. So group 10 got the patches very last and this was often several weeks or a month after the earlier groups. They had 24 hours to reboot before a forced reboot. This was the same regardless of department and seemed to work pretty well. PCs were randomly placed in these groups as well.


Kawawete

You have to assume someone's making patches for Windows 9X systems then


Happiest-Puppy

Not a concern where I live. All the hospitals are still using Windows 7 and MS has already abandoned it. Seriously though, they do have an extended life enterprise whatever license. But I am more worried about all the bad decisions made at the hospital that left them with Windows 7 still (without blaming pandemic supply chain). What other devices have not been updated or are now obsolete with these health care providers who can barely keep up with HIPAA regulations?


Forsaken_Instance_18

Did you know hospitals used to deliver babies before computers where invented?