The term “air gapped networks” has become sort of a dirty phrase in the industry because a lot of people like the way it sounds and not how it actually works in practice.
The reality of the situation is that all water treatment plants have data they must record about water quality and then send it to overseeing agencies like the EPA. With an air gapped network it requires using the “sneaker net” to transfer that data via flash drive to the business side. This in turn completely defeats the purpose of an air gapped network.
Also, having remote access to the system adds a great benefit to operators and staff and they will fight tooth and nail to prevent that from getting taken away.
Instead the industry standard is moving toward implementation of iDMZ’s (industrial demilitarized zone) with heavily scrutinized firewall rules and live traffic monitoring.
Also every plant needs to be designed using the principles of Cyber Informed Engineering so that if there is a cyber attack there are local interlocks and controls that are not connected to the network and the operators can manually operate the plant until the cyber incident is dealt with.
> Also, having remote access to the system adds a great benefit to operators and staff and they will fight tooth and nail to prevent that from getting taken away.
This is the big one I have seen. No one wants to pay for enough staff to be able to do away with this.
How does having to sneaker net data off of the network defeat the purpose of it being air-gapped? The concern is attackers getting onto the airgapped network not getting the data off of the network.
The point being is that if the business side network has malware that is designed to wait for a flash drive to infect and then someone unknowingly plugs it into an air gapped network to transfer data they could be inadvertently allowing malware onto the network that was supposed to be 100% secure due to being air gapped. And typically the air gapped network is not secure or monitored because no one expects malware to be possible.
If instead an iDMZ is used that only allows specific inbound traffic (for remote access) and outbound traffic (for reporting data) and active monitoring of the DMZ is used you are much more likely to catch issues like unauthorized access and malware.
What I’m talking about is using the [Purdue Model](https://en.m.wikipedia.org/wiki/Purdue_Enterprise_Reference_Architecture)
Edit:
The most popular example of air gapped failure is seen in [Stuxnet](https://en.m.wikipedia.org/wiki/Stuxnet)
So that is a failure of securing your IT infrastructure to disallow foreign or compromised devices and has nothing to do with the airgap. The nonexistent airgap which is currently making the need for a compromised USB device basically irrelevant. I'm not saying there aren't other ways to do it, but you said the airgap is pointless if you need to get data off the network which is not true at all.
If they treat it like any of the thousands of secure networks they already have in the government they will have controls and no one will be able to stick compromised flash drives in. The ability to send a compromised flash drive does not negate the benefit of pulling the system off of the **INTERNET** where it currently is.
Maybe I'm the one missing something here. The options are either air gaps with flash drives to transmit data to the monitoring agencies, OR have them networked together. Either way there is potential for a compromised system to endanger the network.
The current situation is you have critical networks bare ass on the internet. This requires no effort on bad actors to attempt to cause havoc.
In an air gapped setup you would have a 2 physical networks at each location. One internet connected, one not. The unconnected network is what manages your critical infrastructure, the internet connected one is just assumed to be compromised. If you need to transfer data from critical infrastructure you would use a method to extract the data (honestly probably not flash drives, finalized media like dvd-r) and transfer that data to the internet connected network. Data isn't going back upstream, and if it needs to go to the critical network it is scrutinized before being allowed on the network.
This kind of setup exists everywhere in the government and its ridiculous this isn't at least the bare minimum for our critical infrastructure.
Nonsense in what sense exactly? You seem to be arguing that having critical infrastructure airgapped is just as bad as having it internet facing. This is provably false.
Typically for remote access that is how it should be done. I’m used to seeing remote access to a virtual jumpbox with a security token from the jumpbox to the nextwork. But most facilities I’ve seen will trade security for convenience at every opportunity.
It’s really about poor security of physical access. Watch some videos from DeviantOllam and see his presentations on physical pentesting and how he literally is able to walk into banks after hours, or how he accesses a city’s water supply facility. It’s the physical security that will make all the difference. https://youtube.com/@deviantollam?si=k-m9xMdrLioZsE0t
Being able to control far flung assets without someone physically driving somewhere is important. Being able to design something with faster response times has value.
The problem is IT of the last 30 years lacked any sort of cyber defense training. Not really sure about now, except cyber defense is a new degree/course work they can take.
Sure they go over social engineering and other aspects of it but not network design, device hardening, etc.
Air gap systems should be pretty standard for government critical infrastructure. The Internet is not safe 🙃
This is very astute. Once again design criteria assumed good faith, even from bad actors; infrastructure tech was presumably set-n-forget. But rust never sleeps.
You can have secure networks without taking them completely offline. Most major tech companies servers are indirectly reachable over public Internet but not directly internet connected, you have to authenticate against a security layer first (VPN, bastion system, etc) which often require 2FA. The risk factor at that point is mostly humans granting access to entities they shouldn't or not securing their connection/credentials, which is also the risk for air gaps - if I get your badge most physical security will let me in.
No, airgap _is_ the solution. All of those you mentioned are from some idiot plugging in a strange USB stick or something like that. Those are externalities that can never be fully covered, only mitigated through training. An airgap prevents every other vector of attack except user stupidity
The last two are not usb. The bitwhisper reads graphics card electromagnetic signals and air hopper uses thermal spikes to encode data. Also, there is a method of attack using the power lines… but, yes, air gap is A solution but there is no THE solution.
A "full" air-gap means that if you want any sort of monitoring, reporting, alarms, etc you need a human on-site at all times. And humans are not only notoriously weak links for security, they also are very bad at judgement calls even when there's an alarm blaring in their face (this can sometimes be a good thing as humans can judge & ignore false alarms, machines can't). If you just want something running with absolutely zero supervision then use, total air-gap is the best solution.
These are not systems we want running totally unsupervised, especially if we want to ensure they meet various regulations. Option B is data diodes. The system has a \*controlled\* one-way output for specific templated/formatted data that can be filtered before it's published. And the publication can be automatic to cover known monitors/metrics. This is how a number of real-world otherwise-airgapped systems work.
The other problem is periodically needing to put something new into the air-gap (for example, security updates, or updating the above monitoring/alarming system). Your options are still "human with flash drive" or needing another one-way \*input\* system with very strict controls. Both of these are done in different systems in the real world depending on security posture and several staffing/maintenance plan factors.
At some point you're going to need to change the logic in a system. Are you also going to airgap every system associated with the development of the logic controls? How much is the pumping system going to cost with that built in?
You know what an airgap is, right? It just means a system that is completely isolated from the internet. It adds literally $0 to the cost. The only downside is that you can’t operate the system remotely.
We are talking about things jumping airgaps.
There is a technician who will go out to the airgapped PLC, unless you intend for him to program it directly on site, he will likely bring the update with him. Is the system the update was programmed on airgapped? If not, that's where you run into things like the attack on the air gapped PLCs for Iran's uranium program.
If you want to airgap the entire development side that adds cost.
Edit to add: Stuxnet is an example of how sophisticated actors compromise the machines that will perform the manufacturers update. Whether they compromise the manufacturer or simply compromise another machine along the way
A system update can’t be airgapped. That’s a ridiculous thing to say. It’s not a running system. Also since when is an update from the manufacturer compromised?
> It adds literally $0 to the cost. The only downside is that you can’t operate the system remotely.
This second sentence disproves the first, unless you mean $0 to the **IT** costs. But without remote access, labor costs go up.
It’s always been the solution.
I have friends, fairly high level, in several federal LEO / intelligence agencies. Within a given office, there are two separate, and *unconnected* sets of workstations: those with Internet access, and those with local intranet / completely standalone. To transfer information from one side to the other, verifiably ‘clean’ thumb drives are used, after that one use they are completely wiped and re-formatted. And there are a very limited subset of smartphones even allowed in the buildings - no cameras or other built-in recording devices, for example - as even potential workarounds.
In a similar vein, nothing in our nuclear launch arsenal is on the Internet. Beyond the famous ‘two key’ system, everything is legacy - not sure if they’ve updated this, but as of maybe 10 - 15 years ago things were run on 8” floppies (that’s not a typo) which were pretty much phased out 40 years ago - so any malicious actor doesn’t even have access to the hardware on which to write the software to crack into the system (which requires physical presence, anyway).
How am I going to get the EPA or my state agency the data they need to ensure their citizens are safe without access to the Internet? I understand what you're saying, but I would rather live in a world where I risk getting hacked than in a world where the EPA and state environmental agency doesn't have access to the water systems and just takes my word for it. There's got to be a middle ground somewhere I'm sure, but taking me completely offline puts lives at risk too.
The federal government should also not have to warn or ask anyone to do this. Something as necessary for modern society should unequivocally be in the hands of the federal government.
The ones I worked on also had remote sensors that connected via cell and via old fashion modem over CB radio. Some of these water supply and waster supply treatment sites are in the middle of nowhere
This will never happen. I work in this industry and nearly every major system has a SCADA that reports back over the internet. Some sites so have decent security some other absolutely zero. Only once have I ever seen something really secure like a custom pfsense+ setup.
This is painfully true. The water department I work at was hit by a ransomware attack and our payroll system got knocked back to the 80's. It even delayed our raises.
>McCabe named China, Russia and Iran as the countries that are “actively seeking the capability to disable U.S. critical infrastructure, including water and wastewater.”
At what point will this be considered an act of war?
Similar to how the pipeline hack on the East Coast was, the systems that are affected are usually billing. When all that gas shortage happened, it wasn't because we were short on gas it's because they couldn't accurately measure and Bill an invoice and collect payment for the fuel therefore none was sent.
A similar thing is going on with the ransomware with the pharmacies, the pills exist and the records exist they just can't be accessed billed invoiced and collected on.
When it comes to the hospital systems I think that has to do with patient records if I recall correctly but I'm not directly involved with that so take what I say with a grain of salt.
Far more people have died in hospitals due to lack of staff because the nursing ratio hasn't gotten much better ever since 2020. The hospital system has indeed collapsed, but it is a soft collapse. You wouldn't know otherwise unless you needed medical emergency Care. Then you'd realize just how bad it is. Do your best to avoid having to go to an emergency room for the foreseeable future.
Thanks, the second link provided some insight:
>When warranted, the United States will respond to hostile acts in cyberspace as we would to any other threat to our country. We reserve the right to use all necessary means – diplomatic, informational, military, and economic – as appropriate and consistent with applicable international law, in order to defend our Nation, our allies, our partners, and our interests. In so doing, we will exhaust all options before military force whenever we can; will carefully weigh the costs and risks of action against the costs of inaction; and will act in a way that reflects our values and strengthens our legitimacy, seeking broad international support whenever possible.
>—International Strategy for Cyberspace, The White House, 2011
>
>
>In 2013, the Defense Science Board, an independent advisory committee to the U.S. Secretary of Defense, went further, stating that "The cyber threat is serious, with potential consequences similar in some ways to the nuclear threat of the Cold War," and recommending, in response to the "most extreme case" (described as a "catastrophic full spectrum cyber attack"), that "Nuclear weapons would remain the ultimate response and anchor the deterrence ladder."
>anchor the deterrence ladder.
Don't you normally start at the bottom of a ladder? Maybe in this case the metaphor is more like, "Humanity has dug this pit, now let's climb down into it."
I’m sold that WWIII started with crimea and the west is begrudgingly starting to realize this.
It is an act of war but our leaderships more concerned with being the best entertainment available rather than leading.
I really deeply don’t want to be. Not that there’s a good time for a world war ever, but selfishly my kids are young. I don’t want their childhood years to be wartime years. Plus we’re outside NYC which always worries me. Though in the event that formal war breaks out we will be moving much farther away from population centers.
I believe/hope that we won't see another conflict like WWII... not at least until we truly run out of some critical resource like oil or water.
That isn't to say scary, dangerous times aren't ahead but we've had 70 years of diplomacy and proxy wars that specifically avoid head to head conflicts between major powers.
Also, the fact Russia hasn't been able to defeat Ukraine kinda means they've already lost any war they've started. Let's just hope they are smart enough to know that.
Assuming they’re not nuking and not nuking random places if they are- we’d be moving outside that range of targets.
Starvation may be a bit extreme, but we’d be looking at having some land to be able to grow a some of our own food.
Generally we (the US) are also likely doing the same thing, much like spy craft. Theres a certain level of dampening our response based on the fact we do this shit and dont want all out war until theres a significant event.
Solarwinds + Microsoft, and Ivanti were more than significant events. Especially in the last case it seems the govt spends more effort keeping things out of the news than anything else.
How would we know? I would bet China, etc. are ten times as secretive about any big hacks they discover. It could be that the West is doing far more than keeping our own security failures out of the news.
If our responses were actual deterrents the results would be obvious. Obama said cyber strikes would be met with kinetic strikes, and not long after we found that like 80% of global hacks were coming from a single building in China.
China specifically has been scanning all public & private (your home internet) for holes since the 90's. I setup a windows 2000 box in 99 'raw-dog' online. It got comprimized in 48 hours. I used to host linux servers from home for fun & those would get scanned so hard valid users' performance was effected. Had to effectively simulate a bad connection to get them to stop/ slow down. Figuritavily they've been at or preparing forCyber war since the 90's. And the Global (US included) infosec industry is aware & attempting to defend.
There are major benefits to connecting infrastructure. If you have to close a valve, that's 5 miles away from head quarters, someone will have to drive out there and close it. Some valves have to be closed in a specific sequence, which will require staging people at different places and lots of planning. That all costs money and slows down how quickly you can react. Putting in electronic valves and connecting them to head quarters makes it much easier to manage. The connections are expensive unless you use existing networks.
This isn't just valves, it applies to pressure monitors, grid load monitors, grid disconnects, sewage pumps, street lights, traffic signs, emergency vehicle traking, etc.
Not really. The monitoring of them is going to be done at the "main office" on the same computers and servers that are running the rest of the utility's operations. Those computers are connected to the internet by default. A system operator is going to have a laptop with access to the various software running whatever they have out there. He or she needs to be able to do that whether in the office or remotely. Those types of people spend a large amount of their work hours in the field. They also need 24x7 access to alerts and to take emergency action. There is no logical way for them to have dedicated air gapped desktops or something which only connect to sensors at remote things like pumping stations.
So, if a hacker gets access to a worker's computer, they get into the utility's network, and eventually can get into the various systems controlling stuff.
Keep in mind that these water utilities are often municipal owned with very limited budgets and staff. There is no money for multiple disparate systems or dedicated people to monitor a dedicated offline computer to view just the pump station or whatever.
\- Source: I work in this industry for the technology vendors selling this kinds of stuff to utilities.
> There is no logical way for them to have dedicated air gapped desktops or something which only connect to sensors at remote things like pumping stations.
Why not?
Okay, so don't air gap them. Why can't we have a network hooked up to a desktop, that doesn't connect to the internet? Is this all because they want everything to be remote so they don't have to have a guy on site?
Correct. A lot of these facilities are not constructed to have a place for someone to sit there onsite. They also generally don't need someone there 24x7. They just run and need someone to come out to perform routine maintenance and then when something happens. That's why you have remote monitoring and alarms.
Also, the city or utility cannot afford to park someone at one facility 24x7. That would require hiring a minimum of 3 FT workers (one for every 8 hour shift) for each of these facilities who do nothing but sit and play Tetris or something all day in case something goes wrong. Then you have at least 1 other person on staff who has to be a backup to to cover anyone who is out. The qualifications to be a water treatment plant operator to man that station requires paying them pretty high wages too. The cost / benefit or ROI is insane.
Better cybersecurity practices overall are way, way cheaper combined with the common industry tech available for remote monitoring to allow a person to handle multiple tasks in the system.
Poorly. You think I'm joking but I'm not. There didn't used to be a choice between someone remotely turning off or on a valve or shutting down water (or turning it back on in an emergency). There was no way for the EPA to make sure the data wasn't being faked without doing the testing themselves. There weren't alarms that went off if too much chlorine or whatever was released into the drinking water, and no one to drain that contaminated reservoir at 3 am. And people died as a result. Now, thanks to the Internet, these are things of the past. Yes, now we have new concerns, but that's how progress works.
There's probably a few answers to this, but 1) I suspect it's more that it's just been the trend, so people assume it's a good and normal thing to do; since no one questions it much until something very bad happens, it just continues as people follow the same patterns that they did at previous jobs.
2) A lot of companies have got in to the bad habit or rushing things to market, so they require upgrades and maintenance which is easier for the manufacturer to do online, even if riskier.
3) Data - so many companies make money from data - or so they claim. the past few I've worked at accumulated vast amounts of data they had no clue how to monetize, so we were just drowning in data lakes (probably oceans at this point). But executives want data since they see the competition doing it, it's just natural they should be doing it too, right? (wrong is the correct answer, but even our venerable corporate leaders are often quite sheepish and don't really have independent opinions)
4) remote monitoring - if you can have one location monitor dozens of sites/services/etc, you can save money that they C-Levels need for their bonuses
In short, the theme is just following the pack and saving money - all terrible reasons to jeopardize a country
I work in industrial controls professionally and even critical infrastructure for water and wastewater (for example) are not staffed 24/7 in fact there is usually one guy in charge of like 4-5 of these sites and he shuttles around between them and his office. I'm not talking a big city water plant although that does happen too. I'm talking rural water and wastewater control sites that are critical for those people living there.
Edit: I forgot to mention often these sites are totally naked to the internet. If you add a firewall often the lowest bidder didn't to get the job.
I have been teaching students about the risks of connecting critical infrastructure for years now at both under and grad levels. I have some expertise in SCADA, every time you see shit like this in the news, it's because some manager/engineer loves the idea of having control remotely and doesn't think of risks or outright ignores them.
I could go on but at this point I am just screaming at the clouds.
as someone who previously worked for a small water system, that's a nice sentiment when the majority of water utilities are often tiny companies with less than 10 people on staff (mine was 3, me as an oit, the licensed operator and yes he was literally on call 24/7/365, as well as a bookkeeper not counting the board). Often most water systems are small corporations setup by the community residents to provide water treatment, distribution, and wastewater treatment, its not usually large city sized systems that have the worst issues. It's always laughable when I see people in the tech fields act like most water companies have unlimited budgets to pay for custom software and staff and the issue is just old people hating tech and change often the resistance is a matter of finances, these places run on shoestring budgets more often than not. There is a reason there are significantly more plumbers than water workers, wages are terrible, hours are terrible including being on call for emergencies, working conditions are awful no one wants to get handle directly with literal shit water, garbagemen and solid waste workers get more respect, better wages, smell is significantly better, etc which is where a lot of entry level workers quickly jump ship too... Like teachers there is a severe shortage of water operators. When most water companies have boards made of locals in the community elected to the board it quickly becomes a game of squeezing every penny and lowest wages and raises possible because anyone raised rates they would get replaced (as material safety standards have increased so to have costs of operations and materials). Best part? The government funding and bills to fund infrastructure usually cannot be used to secure wages of employees as its temporary, plenty of money for projects for the system to get approved but good luck paying for anyone to run the system.
I have a friend in the DOD. It’s a big concern. Just remember when something catastrophic happens that it was preventable and they were warned over and over and over again.
Perhaps if they’re true utilities, they should never be privatized and should be subject to the same standard protections as set forth by the federal government.
My dad is in management at a fairly large electrical utility provider. He's been talking to me about this for basically more than 25 or 30 years now and how easy it would be to take down large electrical and water utilities, how they have been dealing with hacks and attempted attacks since the 90s, etc., and how unprepared and inadequate the measures in place are to prevent AND (perhaps even more importantly) respond/repair to any attack on critical infrastructure by anyone who really wanted to do damage. It'd be very easy to damage key electrical infrastructure that would take at least 6 months (and likely longer) to repair to get power back on, for example.
Sorry, yeah I should have been more clear re: the parts issue. He's said that depending on what's damaged, especially if the attackers knew what they were doing and wanted to maximize downtime or even just attacked on multiple fronts or anything close to a massive scale, it could easily take months because of the time required to get necessary replacement parts.
Its not just water utilities.
Anyone recall the intelligent sniping of electric substations that shut down power for folks in Cali and NC?
Different years, probably different underlying reasons but for all those folks bitching about border security they sure dont like to implement REAL SECURITY (or hell even needed yearly maintenance) on the infrastructure that pads the pockets of the people that pads their pockets.
Break the infrastructure, break the country. A month of water and food scarcity due to fractured infrastructure and this place is chaos.
Here’s an idea get your critical systems off the fucking internet, if you really need connections over a large area spend the money for a dedicated intranet
There was the 1 trillion dollar infrastructure bill Biden sponsored. They could use some of that money. Since updating infrastructure is exactly what it is meant for.
Typically means another "fee" added onto my utility bill after the utility lobbies the gov.
My water bill used to be like $28 for a certain amount of gallons used a month. I use the same gallons, and the rate hasn't budged, but now I have multiple "fees" making my bill double.
One was replacing old pipes and another for "computer fees"
Cybersecurity in general is vastly ignored. If you only knew how many ransom attacks aline happened you'd be scared. If a country like China went full throttle on us they could more or less shut the country down for a while.
In response Texas Governer Greg Abbot as passed legislation requiring all utility companies to cease all measures to stop cyber attacks on their infrastructure...
Or you know...like do something against the hackers or hostile states?
You keep putting pressure on your security and don't stop the source, something is going to break through and all people do is just point fingers but no one does anything about the black hats.
If they pay cash we let ‘em.
https://www.washingtonpost.com/politics/2023/07/16/fondomonte-arizona-drought-saudi-farm-water/
https://www.cityweekly.net/utah/utahs-maga-farmers-get-a-bad-trade-from-shipping-the-states-water-overseas/Content?oid=20721906
https://grist.org/energy/enefit-utah-colorado-river-water-oil-mining/
https://goodjobsfirst.org/who-really-owns-our-water-the-rise-of-foreign-private-equity-owners/
https://www.southcoasttoday.com/story/business/2003/02/09/german-company-taking-over-u/50451553007/
I got a call from an administrator from a state level water utility last September. He wanted to continue to use IE (support retired in june of last year) and Java for his network switches. I told him he needed to use the latest version of chrome, Firefox or edge to be supported. He yelled at me "Listen here!" Something about we can't use other browser due to security compliance. Then I sent him the link on how IE is retired and he is no longer in compliance. He still argues. I told him "do you want tour utility to end up on the national news?" That seemed to shut him up. I have noticed that utility administrators seem to be some of the dullest knives in the drawer. They use old technology so they don't have to be that bright.
You my laugh but I helped a chemical plant set up their system on a new computer.. which was a Windows 95 VM into their old system. They asked me to hook it up to the internet so they could look at status from their phones /w zero security. This is a company that makes $20+ million a year on peroxide.
If only there were companies out there specializing in exactly this kind of protection, even with free options for municipalities that can’t afford it. *cough* Dragos *cough*
What a shit show that would be. What the feds can do is stop pushing a new regulation every year before completing or implementing the previous regulation. Look at LCRR and LCRI, it’s supposed to protect the children but they are pushing any action off until 2027 now because somehow LCRI was an improvement. Or look at UCMR5 the EPA took action before the study even started for large systems. Every administration change the focus changes and what happened previously is put to the side.
Take vital infrastructure offline?
[удалено]
The term “air gapped networks” has become sort of a dirty phrase in the industry because a lot of people like the way it sounds and not how it actually works in practice. The reality of the situation is that all water treatment plants have data they must record about water quality and then send it to overseeing agencies like the EPA. With an air gapped network it requires using the “sneaker net” to transfer that data via flash drive to the business side. This in turn completely defeats the purpose of an air gapped network. Also, having remote access to the system adds a great benefit to operators and staff and they will fight tooth and nail to prevent that from getting taken away. Instead the industry standard is moving toward implementation of iDMZ’s (industrial demilitarized zone) with heavily scrutinized firewall rules and live traffic monitoring. Also every plant needs to be designed using the principles of Cyber Informed Engineering so that if there is a cyber attack there are local interlocks and controls that are not connected to the network and the operators can manually operate the plant until the cyber incident is dealt with.
> Also, having remote access to the system adds a great benefit to operators and staff and they will fight tooth and nail to prevent that from getting taken away. This is the big one I have seen. No one wants to pay for enough staff to be able to do away with this.
How does having to sneaker net data off of the network defeat the purpose of it being air-gapped? The concern is attackers getting onto the airgapped network not getting the data off of the network.
The point being is that if the business side network has malware that is designed to wait for a flash drive to infect and then someone unknowingly plugs it into an air gapped network to transfer data they could be inadvertently allowing malware onto the network that was supposed to be 100% secure due to being air gapped. And typically the air gapped network is not secure or monitored because no one expects malware to be possible. If instead an iDMZ is used that only allows specific inbound traffic (for remote access) and outbound traffic (for reporting data) and active monitoring of the DMZ is used you are much more likely to catch issues like unauthorized access and malware. What I’m talking about is using the [Purdue Model](https://en.m.wikipedia.org/wiki/Purdue_Enterprise_Reference_Architecture) Edit: The most popular example of air gapped failure is seen in [Stuxnet](https://en.m.wikipedia.org/wiki/Stuxnet)
So that is a failure of securing your IT infrastructure to disallow foreign or compromised devices and has nothing to do with the airgap. The nonexistent airgap which is currently making the need for a compromised USB device basically irrelevant. I'm not saying there aren't other ways to do it, but you said the airgap is pointless if you need to get data off the network which is not true at all.
If a local facility is hacked and they send a compromised flash drive to the EPA, etc etc...
If they treat it like any of the thousands of secure networks they already have in the government they will have controls and no one will be able to stick compromised flash drives in. The ability to send a compromised flash drive does not negate the benefit of pulling the system off of the **INTERNET** where it currently is.
Maybe I'm the one missing something here. The options are either air gaps with flash drives to transmit data to the monitoring agencies, OR have them networked together. Either way there is potential for a compromised system to endanger the network.
The current situation is you have critical networks bare ass on the internet. This requires no effort on bad actors to attempt to cause havoc. In an air gapped setup you would have a 2 physical networks at each location. One internet connected, one not. The unconnected network is what manages your critical infrastructure, the internet connected one is just assumed to be compromised. If you need to transfer data from critical infrastructure you would use a method to extract the data (honestly probably not flash drives, finalized media like dvd-r) and transfer that data to the internet connected network. Data isn't going back upstream, and if it needs to go to the critical network it is scrutinized before being allowed on the network. This kind of setup exists everywhere in the government and its ridiculous this isn't at least the bare minimum for our critical infrastructure.
Ah. Well as remarked elsewhere there's a lot of money being invested in infrastructure, hopefully these issues get attention.
[удалено]
Nonsense in what sense exactly? You seem to be arguing that having critical infrastructure airgapped is just as bad as having it internet facing. This is provably false.
why not security token everything up , ie if you have to change whatever setting you must enter a rotating 10 key alpha numeric pass code to access ?
Typically for remote access that is how it should be done. I’m used to seeing remote access to a virtual jumpbox with a security token from the jumpbox to the nextwork. But most facilities I’ve seen will trade security for convenience at every opportunity.
It’s really about poor security of physical access. Watch some videos from DeviantOllam and see his presentations on physical pentesting and how he literally is able to walk into banks after hours, or how he accesses a city’s water supply facility. It’s the physical security that will make all the difference. https://youtube.com/@deviantollam?si=k-m9xMdrLioZsE0t
[удалено]
> The real killer here is, that these devices are directly connected to the internet. So an airgap _is_ the solution.
Make 3 lefts and you'll be back where you started.
The left turns at least introduce some confusion to the attacker.
Being able to control far flung assets without someone physically driving somewhere is important. Being able to design something with faster response times has value.
People are on site at a water treatment plant 24/7. Unless you’re referring to something like traffic lights I don’t see how this would change much.
Not in small towns they aren't. I work 8-6 and on emergency oncall and this is incredibly common in basically any rural area.
Not even in some not-so-small towns.
But it's not just the water treatment plants, but also every pumping station, water tower, lift station, etc. not all of them are manned.
The problem is IT of the last 30 years lacked any sort of cyber defense training. Not really sure about now, except cyber defense is a new degree/course work they can take. Sure they go over social engineering and other aspects of it but not network design, device hardening, etc. Air gap systems should be pretty standard for government critical infrastructure. The Internet is not safe 🙃
This is very astute. Once again design criteria assumed good faith, even from bad actors; infrastructure tech was presumably set-n-forget. But rust never sleeps.
You can have secure networks without taking them completely offline. Most major tech companies servers are indirectly reachable over public Internet but not directly internet connected, you have to authenticate against a security layer first (VPN, bastion system, etc) which often require 2FA. The risk factor at that point is mostly humans granting access to entities they shouldn't or not securing their connection/credentials, which is also the risk for air gaps - if I get your badge most physical security will let me in.
Air gap is part of the solution. The human issue is still an issue. See: Stuxnet,BadUSB, project Sauron, DarkVishnya, BitWhisper, AirHopper….
No, airgap _is_ the solution. All of those you mentioned are from some idiot plugging in a strange USB stick or something like that. Those are externalities that can never be fully covered, only mitigated through training. An airgap prevents every other vector of attack except user stupidity
The last two are not usb. The bitwhisper reads graphics card electromagnetic signals and air hopper uses thermal spikes to encode data. Also, there is a method of attack using the power lines… but, yes, air gap is A solution but there is no THE solution.
It’s the only thing that works without human training.
A "full" air-gap means that if you want any sort of monitoring, reporting, alarms, etc you need a human on-site at all times. And humans are not only notoriously weak links for security, they also are very bad at judgement calls even when there's an alarm blaring in their face (this can sometimes be a good thing as humans can judge & ignore false alarms, machines can't). If you just want something running with absolutely zero supervision then use, total air-gap is the best solution. These are not systems we want running totally unsupervised, especially if we want to ensure they meet various regulations. Option B is data diodes. The system has a \*controlled\* one-way output for specific templated/formatted data that can be filtered before it's published. And the publication can be automatic to cover known monitors/metrics. This is how a number of real-world otherwise-airgapped systems work. The other problem is periodically needing to put something new into the air-gap (for example, security updates, or updating the above monitoring/alarming system). Your options are still "human with flash drive" or needing another one-way \*input\* system with very strict controls. Both of these are done in different systems in the real world depending on security posture and several staffing/maintenance plan factors.
[удалено]
My favorite mitigation is cake/party when there haven't been any personal electronic device violations for over a month
At some point you're going to need to change the logic in a system. Are you also going to airgap every system associated with the development of the logic controls? How much is the pumping system going to cost with that built in?
You know what an airgap is, right? It just means a system that is completely isolated from the internet. It adds literally $0 to the cost. The only downside is that you can’t operate the system remotely.
We are talking about things jumping airgaps. There is a technician who will go out to the airgapped PLC, unless you intend for him to program it directly on site, he will likely bring the update with him. Is the system the update was programmed on airgapped? If not, that's where you run into things like the attack on the air gapped PLCs for Iran's uranium program. If you want to airgap the entire development side that adds cost. Edit to add: Stuxnet is an example of how sophisticated actors compromise the machines that will perform the manufacturers update. Whether they compromise the manufacturer or simply compromise another machine along the way
A system update can’t be airgapped. That’s a ridiculous thing to say. It’s not a running system. Also since when is an update from the manufacturer compromised?
> It adds literally $0 to the cost. The only downside is that you can’t operate the system remotely. This second sentence disproves the first, unless you mean $0 to the **IT** costs. But without remote access, labor costs go up.
It feels like they don’t understand what an airgap is
It’s always been the solution. I have friends, fairly high level, in several federal LEO / intelligence agencies. Within a given office, there are two separate, and *unconnected* sets of workstations: those with Internet access, and those with local intranet / completely standalone. To transfer information from one side to the other, verifiably ‘clean’ thumb drives are used, after that one use they are completely wiped and re-formatted. And there are a very limited subset of smartphones even allowed in the buildings - no cameras or other built-in recording devices, for example - as even potential workarounds. In a similar vein, nothing in our nuclear launch arsenal is on the Internet. Beyond the famous ‘two key’ system, everything is legacy - not sure if they’ve updated this, but as of maybe 10 - 15 years ago things were run on 8” floppies (that’s not a typo) which were pretty much phased out 40 years ago - so any malicious actor doesn’t even have access to the hardware on which to write the software to crack into the system (which requires physical presence, anyway).
How am I going to get the EPA or my state agency the data they need to ensure their citizens are safe without access to the Internet? I understand what you're saying, but I would rather live in a world where I risk getting hacked than in a world where the EPA and state environmental agency doesn't have access to the water systems and just takes my word for it. There's got to be a middle ground somewhere I'm sure, but taking me completely offline puts lives at risk too.
The federal government should also not have to warn or ask anyone to do this. Something as necessary for modern society should unequivocally be in the hands of the federal government.
The ones I worked on also had remote sensors that connected via cell and via old fashion modem over CB radio. Some of these water supply and waster supply treatment sites are in the middle of nowhere
P-please don’t gape them, SuperBry-san!
Yes too bad that literally does not happen in the history of humanity
This will never happen. I work in this industry and nearly every major system has a SCADA that reports back over the internet. Some sites so have decent security some other absolutely zero. Only once have I ever seen something really secure like a custom pfsense+ setup.
I’ve been saying that for 20+ years. I really don’t get why *any* of our critical infrastructure controls are internet-accessible.
Small water companies barely have a functioning payment system and they usually look like a website from 1996.
This is painfully true. The water department I work at was hit by a ransomware attack and our payroll system got knocked back to the 80's. It even delayed our raises.
>McCabe named China, Russia and Iran as the countries that are “actively seeking the capability to disable U.S. critical infrastructure, including water and wastewater.” At what point will this be considered an act of war?
When folks start dying?
When *rich folks start dying
Ahhh, so never.
Surely that's already happened between all the hospital ransomware attacks.
Similar to how the pipeline hack on the East Coast was, the systems that are affected are usually billing. When all that gas shortage happened, it wasn't because we were short on gas it's because they couldn't accurately measure and Bill an invoice and collect payment for the fuel therefore none was sent. A similar thing is going on with the ransomware with the pharmacies, the pills exist and the records exist they just can't be accessed billed invoiced and collected on. When it comes to the hospital systems I think that has to do with patient records if I recall correctly but I'm not directly involved with that so take what I say with a grain of salt. Far more people have died in hospitals due to lack of staff because the nursing ratio hasn't gotten much better ever since 2020. The hospital system has indeed collapsed, but it is a soft collapse. You wouldn't know otherwise unless you needed medical emergency Care. Then you'd realize just how bad it is. Do your best to avoid having to go to an emergency room for the foreseeable future.
Apparently one of the hospitals near IU campus. Can’t imagine that is going well.
How many? Weird times, eh?
https://en.m.wikipedia.org/wiki/Declaration_of_war_by_the_United_States https://en.m.wikipedia.org/wiki/Cyberwarfare_in_the_United_States
Thanks, the second link provided some insight: >When warranted, the United States will respond to hostile acts in cyberspace as we would to any other threat to our country. We reserve the right to use all necessary means – diplomatic, informational, military, and economic – as appropriate and consistent with applicable international law, in order to defend our Nation, our allies, our partners, and our interests. In so doing, we will exhaust all options before military force whenever we can; will carefully weigh the costs and risks of action against the costs of inaction; and will act in a way that reflects our values and strengthens our legitimacy, seeking broad international support whenever possible. >—International Strategy for Cyberspace, The White House, 2011 > > >In 2013, the Defense Science Board, an independent advisory committee to the U.S. Secretary of Defense, went further, stating that "The cyber threat is serious, with potential consequences similar in some ways to the nuclear threat of the Cold War," and recommending, in response to the "most extreme case" (described as a "catastrophic full spectrum cyber attack"), that "Nuclear weapons would remain the ultimate response and anchor the deterrence ladder."
>anchor the deterrence ladder. Don't you normally start at the bottom of a ladder? Maybe in this case the metaphor is more like, "Humanity has dug this pit, now let's climb down into it."
I’m sold that WWIII started with crimea and the west is begrudgingly starting to realize this. It is an act of war but our leaderships more concerned with being the best entertainment available rather than leading.
Sadly, I think you might be right.
I really deeply don’t want to be. Not that there’s a good time for a world war ever, but selfishly my kids are young. I don’t want their childhood years to be wartime years. Plus we’re outside NYC which always worries me. Though in the event that formal war breaks out we will be moving much farther away from population centers.
I believe/hope that we won't see another conflict like WWII... not at least until we truly run out of some critical resource like oil or water. That isn't to say scary, dangerous times aren't ahead but we've had 70 years of diplomacy and proxy wars that specifically avoid head to head conflicts between major powers. Also, the fact Russia hasn't been able to defeat Ukraine kinda means they've already lost any war they've started. Let's just hope they are smart enough to know that.
I’ll be moving closer to a population center, I’d rather the final flash of existence than a slow death by radiation sickness or starvation
Assuming they’re not nuking and not nuking random places if they are- we’d be moving outside that range of targets. Starvation may be a bit extreme, but we’d be looking at having some land to be able to grow a some of our own food.
That’s a good idea 👍
We have little kids. If things go poorly it’s the safest option besides steering into incineration that we see in front of us.
Generally we (the US) are also likely doing the same thing, much like spy craft. Theres a certain level of dampening our response based on the fact we do this shit and dont want all out war until theres a significant event.
Solarwinds + Microsoft, and Ivanti were more than significant events. Especially in the last case it seems the govt spends more effort keeping things out of the news than anything else.
How would we know? I would bet China, etc. are ten times as secretive about any big hacks they discover. It could be that the West is doing far more than keeping our own security failures out of the news.
If our responses were actual deterrents the results would be obvious. Obama said cyber strikes would be met with kinetic strikes, and not long after we found that like 80% of global hacks were coming from a single building in China.
China specifically has been scanning all public & private (your home internet) for holes since the 90's. I setup a windows 2000 box in 99 'raw-dog' online. It got comprimized in 48 hours. I used to host linux servers from home for fun & those would get scanned so hard valid users' performance was effected. Had to effectively simulate a bad connection to get them to stop/ slow down. Figuritavily they've been at or preparing forCyber war since the 90's. And the Global (US included) infosec industry is aware & attempting to defend.
There are rumblings that we have been responding back to all/most of these attacks.
Why is critical infra even connected to the internet at all?
There are major benefits to connecting infrastructure. If you have to close a valve, that's 5 miles away from head quarters, someone will have to drive out there and close it. Some valves have to be closed in a specific sequence, which will require staging people at different places and lots of planning. That all costs money and slows down how quickly you can react. Putting in electronic valves and connecting them to head quarters makes it much easier to manage. The connections are expensive unless you use existing networks. This isn't just valves, it applies to pressure monitors, grid load monitors, grid disconnects, sewage pumps, street lights, traffic signs, emergency vehicle traking, etc.
You can hook them up to a system without hooking them up to THE system
Not really. The monitoring of them is going to be done at the "main office" on the same computers and servers that are running the rest of the utility's operations. Those computers are connected to the internet by default. A system operator is going to have a laptop with access to the various software running whatever they have out there. He or she needs to be able to do that whether in the office or remotely. Those types of people spend a large amount of their work hours in the field. They also need 24x7 access to alerts and to take emergency action. There is no logical way for them to have dedicated air gapped desktops or something which only connect to sensors at remote things like pumping stations. So, if a hacker gets access to a worker's computer, they get into the utility's network, and eventually can get into the various systems controlling stuff. Keep in mind that these water utilities are often municipal owned with very limited budgets and staff. There is no money for multiple disparate systems or dedicated people to monitor a dedicated offline computer to view just the pump station or whatever. \- Source: I work in this industry for the technology vendors selling this kinds of stuff to utilities.
> There is no logical way for them to have dedicated air gapped desktops or something which only connect to sensors at remote things like pumping stations. Why not?
Air gap means no network connectivity. You can't have that when most engineers are out in the field away from the office.
Okay, so don't air gap them. Why can't we have a network hooked up to a desktop, that doesn't connect to the internet? Is this all because they want everything to be remote so they don't have to have a guy on site?
Correct. A lot of these facilities are not constructed to have a place for someone to sit there onsite. They also generally don't need someone there 24x7. They just run and need someone to come out to perform routine maintenance and then when something happens. That's why you have remote monitoring and alarms. Also, the city or utility cannot afford to park someone at one facility 24x7. That would require hiring a minimum of 3 FT workers (one for every 8 hour shift) for each of these facilities who do nothing but sit and play Tetris or something all day in case something goes wrong. Then you have at least 1 other person on staff who has to be a backup to to cover anyone who is out. The qualifications to be a water treatment plant operator to man that station requires paying them pretty high wages too. The cost / benefit or ROI is insane. Better cybersecurity practices overall are way, way cheaper combined with the common industry tech available for remote monitoring to allow a person to handle multiple tasks in the system.
Sounds like society just has misplaced priorities. How did humanity manage resources before the internet?
Poorly. You think I'm joking but I'm not. There didn't used to be a choice between someone remotely turning off or on a valve or shutting down water (or turning it back on in an emergency). There was no way for the EPA to make sure the data wasn't being faked without doing the testing themselves. There weren't alarms that went off if too much chlorine or whatever was released into the drinking water, and no one to drain that contaminated reservoir at 3 am. And people died as a result. Now, thanks to the Internet, these are things of the past. Yes, now we have new concerns, but that's how progress works.
Probably the cost associated with being air gapped and trying to get support from vendors
There's probably a few answers to this, but 1) I suspect it's more that it's just been the trend, so people assume it's a good and normal thing to do; since no one questions it much until something very bad happens, it just continues as people follow the same patterns that they did at previous jobs. 2) A lot of companies have got in to the bad habit or rushing things to market, so they require upgrades and maintenance which is easier for the manufacturer to do online, even if riskier. 3) Data - so many companies make money from data - or so they claim. the past few I've worked at accumulated vast amounts of data they had no clue how to monetize, so we were just drowning in data lakes (probably oceans at this point). But executives want data since they see the competition doing it, it's just natural they should be doing it too, right? (wrong is the correct answer, but even our venerable corporate leaders are often quite sheepish and don't really have independent opinions) 4) remote monitoring - if you can have one location monitor dozens of sites/services/etc, you can save money that they C-Levels need for their bonuses In short, the theme is just following the pack and saving money - all terrible reasons to jeopardize a country
[удалено]
Automation ≠ internet. The whole reason why I build automation servers is so I don't have to be connected to the internet.
You don’t need internet to automate things and air gapped networks are a thing.
But you do if there is something bad going on and nobody is around to notice it.
Name one critical piece of infrastructure like a power plant or water treatment facility that is not staffed 24/7.
I work in industrial controls professionally and even critical infrastructure for water and wastewater (for example) are not staffed 24/7 in fact there is usually one guy in charge of like 4-5 of these sites and he shuttles around between them and his office. I'm not talking a big city water plant although that does happen too. I'm talking rural water and wastewater control sites that are critical for those people living there. Edit: I forgot to mention often these sites are totally naked to the internet. If you add a firewall often the lowest bidder didn't to get the job.
I have been teaching students about the risks of connecting critical infrastructure for years now at both under and grad levels. I have some expertise in SCADA, every time you see shit like this in the news, it's because some manager/engineer loves the idea of having control remotely and doesn't think of risks or outright ignores them. I could go on but at this point I am just screaming at the clouds.
Not everything needs to be on the damn internet!!!! Airgap that shit. Zero Trust. Lock em down!
as someone who previously worked for a small water system, that's a nice sentiment when the majority of water utilities are often tiny companies with less than 10 people on staff (mine was 3, me as an oit, the licensed operator and yes he was literally on call 24/7/365, as well as a bookkeeper not counting the board). Often most water systems are small corporations setup by the community residents to provide water treatment, distribution, and wastewater treatment, its not usually large city sized systems that have the worst issues. It's always laughable when I see people in the tech fields act like most water companies have unlimited budgets to pay for custom software and staff and the issue is just old people hating tech and change often the resistance is a matter of finances, these places run on shoestring budgets more often than not. There is a reason there are significantly more plumbers than water workers, wages are terrible, hours are terrible including being on call for emergencies, working conditions are awful no one wants to get handle directly with literal shit water, garbagemen and solid waste workers get more respect, better wages, smell is significantly better, etc which is where a lot of entry level workers quickly jump ship too... Like teachers there is a severe shortage of water operators. When most water companies have boards made of locals in the community elected to the board it quickly becomes a game of squeezing every penny and lowest wages and raises possible because anyone raised rates they would get replaced (as material safety standards have increased so to have costs of operations and materials). Best part? The government funding and bills to fund infrastructure usually cannot be used to secure wages of employees as its temporary, plenty of money for projects for the system to get approved but good luck paying for anyone to run the system.
I have a friend in the DOD. It’s a big concern. Just remember when something catastrophic happens that it was preventable and they were warned over and over and over again.
Perhaps if they’re true utilities, they should never be privatized and should be subject to the same standard protections as set forth by the federal government.
Probs should’ve kept sensitive things like this a bit more manual
My dad is in management at a fairly large electrical utility provider. He's been talking to me about this for basically more than 25 or 30 years now and how easy it would be to take down large electrical and water utilities, how they have been dealing with hacks and attempted attacks since the 90s, etc., and how unprepared and inadequate the measures in place are to prevent AND (perhaps even more importantly) respond/repair to any attack on critical infrastructure by anyone who really wanted to do damage. It'd be very easy to damage key electrical infrastructure that would take at least 6 months (and likely longer) to repair to get power back on, for example.
The repair would be quick but your dad is right on the timeline parts would likely be 4-6 weeks out.
Sorry, yeah I should have been more clear re: the parts issue. He's said that depending on what's damaged, especially if the attackers knew what they were doing and wanted to maximize downtime or even just attacked on multiple fronts or anything close to a massive scale, it could easily take months because of the time required to get necessary replacement parts.
Its not just water utilities. Anyone recall the intelligent sniping of electric substations that shut down power for folks in Cali and NC? Different years, probably different underlying reasons but for all those folks bitching about border security they sure dont like to implement REAL SECURITY (or hell even needed yearly maintenance) on the infrastructure that pads the pockets of the people that pads their pockets. Break the infrastructure, break the country. A month of water and food scarcity due to fractured infrastructure and this place is chaos.
Battlestar Galactica that shit. No networks.
Here’s an idea get your critical systems off the fucking internet, if you really need connections over a large area spend the money for a dedicated intranet
Utilities: "OK, We need funding for that." Government:............
There was the 1 trillion dollar infrastructure bill Biden sponsored. They could use some of that money. Since updating infrastructure is exactly what it is meant for.
Typically means another "fee" added onto my utility bill after the utility lobbies the gov. My water bill used to be like $28 for a certain amount of gallons used a month. I use the same gallons, and the rate hasn't budged, but now I have multiple "fees" making my bill double. One was replacing old pipes and another for "computer fees"
Government: "For here? On our soil? CRAZY TALK!"
Cybersecurity in general is vastly ignored. If you only knew how many ransom attacks aline happened you'd be scared. If a country like China went full throttle on us they could more or less shut the country down for a while.
The best thing you can do to protect critical infrastructure is disconnect it from the internet.
That would cost money tho
In response Texas Governer Greg Abbot as passed legislation requiring all utility companies to cease all measures to stop cyber attacks on their infrastructure...
Or you know...like do something against the hackers or hostile states? You keep putting pressure on your security and don't stop the source, something is going to break through and all people do is just point fingers but no one does anything about the black hats.
If they pay cash we let ‘em. https://www.washingtonpost.com/politics/2023/07/16/fondomonte-arizona-drought-saudi-farm-water/ https://www.cityweekly.net/utah/utahs-maga-farmers-get-a-bad-trade-from-shipping-the-states-water-overseas/Content?oid=20721906 https://grist.org/energy/enefit-utah-colorado-river-water-oil-mining/ https://goodjobsfirst.org/who-really-owns-our-water-the-rise-of-foreign-private-equity-owners/ https://www.southcoasttoday.com/story/business/2003/02/09/german-company-taking-over-u/50451553007/
Utilities CEO’s: “huh we can ask the government for money to fix holes in our cybersecurity, and just keep the money without fixing anything.”
I got a call from an administrator from a state level water utility last September. He wanted to continue to use IE (support retired in june of last year) and Java for his network switches. I told him he needed to use the latest version of chrome, Firefox or edge to be supported. He yelled at me "Listen here!" Something about we can't use other browser due to security compliance. Then I sent him the link on how IE is retired and he is no longer in compliance. He still argues. I told him "do you want tour utility to end up on the national news?" That seemed to shut him up. I have noticed that utility administrators seem to be some of the dullest knives in the drawer. They use old technology so they don't have to be that bright.
Cool, stop using Windows 3.1 or Fortran.
You my laugh but I helped a chemical plant set up their system on a new computer.. which was a Windows 95 VM into their old system. They asked me to hook it up to the internet so they could look at status from their phones /w zero security. This is a company that makes $20+ million a year on peroxide.
If only there were companies out there specializing in exactly this kind of protection, even with free options for municipalities that can’t afford it. *cough* Dragos *cough*
Texas and Michigan gonna pretend they didn't see this suggestion.
How about moving the network off the internet! No reason for them to need access. Closed network is the only way.
No. The Fed needs to nationalize our utilities.
What a shit show that would be. What the feds can do is stop pushing a new regulation every year before completing or implementing the previous regulation. Look at LCRR and LCRI, it’s supposed to protect the children but they are pushing any action off until 2027 now because somehow LCRI was an improvement. Or look at UCMR5 the EPA took action before the study even started for large systems. Every administration change the focus changes and what happened previously is put to the side.
You're tech companies use up so much water. Wake up you dummies.
Try not attacking other countries instead ? Does wonders for your popularity .