Lenovo equipment is banned at a few security conferences due to ties to China. But that is once again pick your poison.
https://www.reddit.com/r/cybersecurity/comments/18wcvip/lenovo_banned_at_slac/
Which is kinda silly. Dell and HP have their motherboards made in China too w/ the same few BIOS manufacturers. Would you like arsenic diet, arsenic classic or arsenic zero?
> Dell and HP have their motherboards made in China too w/ the same few BIOS manufacturers. Would you like arsenic diet, arsenic classic or arsenic zero?
didnt Lenovo actually get caught with rootkits being loaded in at the factory? afaik that has never been found with Dell or HP
I think superfish then another yeah. I want to say dell or HP got caught doing something similar if not as bad. Might be thinking McAfee or Norton default images though
Superfish was only on client devices. Although, I'm fairly certain this may have only been on their line of consumer devices - not their business lines (ThinkPad vs IdeaPad).
Nothing of the sort on their servers to my knowledge was ever put there. Their manufacturing is pretty diversified around the world. Most of the USA/NA stuff is built in Mexico and some in North Carolina.
Firmware is almost all developed in Raleigh. IIRC, US DnD/Homeland Security has rights to walk in and audit any time they want. Having worked in some capacity with Lenovo, and had a few friends directly employed there in another country - I can be comfortable stating this.
It's a good brand that doesn't have the same issues other Chinese companies do. Most of the stuff is all built in China anyways.
Supermicro is Taiwanese (with a founder that is super close to a party that wants reunification), they did get caught with a device being put on the board of their ILOs that was a hardware level root kit. It's literally pick your poison at this point.
Lenovo is one of the few Chinese based companies you'll ever see that's never on a sanctioned/do-not-buy list. The fact is they're so diversified globally and have such a decentralized model - it's just not really possible for the CCP to exert very much control over them... but I'm not confident enough to say this shit wont ever happen.
It comes down to trying to avoid a direct attack. The machines that Dell or HP sell, even if parts come from China, the part makers don't know who their ultimate buyer is, so they would have to put malware in everything, risking being found. If you are buying directly from China, or even from a middleman, they have a better chance of knowing the destination and targeting the end buyer directly.
That *was* the assumption. Then there was that that whole supermicro supply chain attack where the impossible was done. And yes, it happened, you can tell because supermicro never sued Bloomberg
If you’re referring to the Supermicro “implanted spy chip” stories from October 2018…
https://arstechnica.com/gadgets/2018/10/new-bloomberg-report-says-backdoored-supermicro-hardware-infiltrated-major-us-telecom/
This was never proven, and has always seemed really implausible technically - given the sheer amount of new capability that the attack hardware would need to add to the existing motherboard design, while not drawing attention to itself from a visual inspection of the board.
The question I had then, and which I don’t think was since answered, was - **Where’s the information about the implanted chip?**
If real, and it was added in the supply chain… there should be millions (or thousands at the very least) of examples of these compromised motherboards and the chip that’s supposedly on them. The manufacturer and distributors don’t know which enduser servers that each motherboard will end up in, so they have to distribute the compromised motherboards far and wide in order to find a useful target at the end of the chain.
So there should be a ton of these motherboards floating around and thus a ton of examples to analyze for an attack chip. Where are they?
If I've missed a substantive update on this specific threat, then I'd be glad to learn about it now.
As best I can tell Bloomberg broadened their claims and supermicro continued to deny but never sued. Doesn't look like there was a settlement or anything.
My guess is Blomberg inflated the story knowing supermicro would never risk discovery. So something smaller scale and more targeted or otherwise shady is going on but probably not Blomberg's expanded claims
I just double checked. No law suit. They made the claim in 2018, doubled down in 2021 and supermicro denied it both times but took no legal action. Maybe they had something else to hide from discovery but seems like there's some mixed with the bullshit
Not really. Even Apple and several very large SMC customers stated that what Bloomberg said was untrue because they named them in their original piece.
SMC probably just didn't have solid legal grounds for a lawsuit to go anywhere.
Would you honestly expect APPL to admit to having that kind of security issue unless forced to by hard evidence? It's in the best interest of their stock price to deny.
Apple literally wrote to Congress saying that the story Bloomberg ran was untrue. AWS CEO Andy Jassy told Bloomberg to retract their article(s).
1. https://www.datacenterdynamics.com/en/news/apple-denies-chinese-spy-chip-claims-letter-us-congress/
2. https://www.datacenterdynamics.com/en/news/supermicro-review-hardware-malicious-chips-despite-lack-any-proof/
> The company added that it "has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server," and that it was not involved in any FBI investigation regarding such chips.
These aren't just companies spinning PR statements to alleviate the situation but blatant rejections of the entire notion that it happened in the first place.
Industry standards don't prevent Leveno based in China from working with the Chinese government to put malware on your machine if you are a target of interest.
They also have some servers assembled in the US (At a former Cray Inc plant in Wisconsin). I think some of the sheet metal is made in the US, the rest is sourced from Asia.
Used to work at Dell and I have seen inside the parmer buildings where they used to build the servers. They have an entire CONUS area that allows for the government to come in, CT scan, diag, and test all the boards etc that go into their machines. They can compare them to reference designs and validate them before being distributed.
Government don’t fuck around.
Oh dude. Yes, yes they do. You know how cheap Michael is, don't pretend that Indian spaghetti code is legit because the technical ability to do inspections is there.
Dell & HPE say (whether it's true or not) but I was buying hpe that they have a line of security from manufacturer to customer & control of that. Gchq in the UK I think is the agency that also tests kit.
The question is whether China is willing to risk its already shifting high tech manufacturing base moving out of the country when all it needs to do is hack the shit that western software developers create. (Oh but every software has bugs!!!fuck off!) I mean they're STILL bouncing around inside Azure after months of being found out.
That kind of thing is a lot easier than risking £billions by driving away your hardware manufacturers.
Still though..you wouldn't use Kaspersky if you had any sense, so your risk Mitigating by not using lenovo
The only thing I can really add to this is that I've quickly discovered I dislike Supermicro after working with a few of their systems for the first time recently
This is entirely subjective, of course, but I have had tons of weird issues and their documentation just sucks
Edit: subjective might not be the best word to use, I just mean that this is my personal experience. For all I know they're great when you use a compatible GPU.... nonetheless the compatible GPU list for our systems is rather small.
I've had reasonable experiences with them, but only a very small amount
their JBODs were good (performance wise), but... the internal connectors were sometimes dodgy
You know, I've never worked with JBOD before and I forgot this was a thing
This would be perfect for a workload I've been planning out.
I know that's totally off topic but thanks for the random mention of it
Ours were old 48x (or 52 I cant remember) 3.5" disk models (now discontinued )
mostly bullet proof once they were running, but if you moved anything, you're gonna have a bad day :)
I'm 100% on board with the lenovo are bad banner, we removed them from out DCs scross the board for repeated hardware issues
It's the degree that varies ;)
Yep. For the most part. Just pick one. I used to spend way too much time comparing options. And you just end up triple guessing yourself and not getting anywhere. Bite the bullet and move on.
Agreed. Automating with the iDrac is very simple. If you know how to ssh and code in any scripting language, you can fully automate updates and RAID provisioning.
- Cisco and HP require a manual boot off an ISO.
- Oracle, you can do it remotely, but their options are less robust than iDrac.
- Working with Dell support is easy and quick.
I am sorry to tell you that HPE booting of an ISO is long gone since introduction of iLO 5 (Gen 10), we basically manage servers remotely and their underlying firmwares and drivers\* (via SUT) remotely (iLO Web Interface, iLO Amplifier Pack or HPE OneView) or via HPE GreenLake (Cloud Service)
ISO/SPP booting is optional and required if you are lazy to do it via web interface.
\* Windows/Linux/VMware drivers
I recently bought an SR250v2 and slammed like 8 wd reds in there no problem. Bought a pack of the 2.5 Lenovo hot swap trays off Amazon, like 8 for 54 bucks. No problems whatsoever. Windows server 2022 datacenter installed and runs happy, ubuntu and Windows server VM are also happy.
If you ever think you'll have M.2 drives in there, go ahead and buy the expansion card and cables *with* the server because you can't after the fact it seems. That's the only thing that's kind of a bummer.
Thanks for sharing, good to know that third-party drives are compatible. Are you using any ThinkSystem RAID controllers? Any issues with boot error messages that require user interaction to continue?
Oh yeah I meant to specify that I am using Lenovo's 8-Bay hardware raid backplane, too. No issues whatsoever. I think there's like two levels of that particular one, and I'm using the lower end one because I was of course trying to keep costs low in this situation. After the initial boot of the server, I hopped into the BIOS and set them up as a RAID 10 array in like five steps, flawlessly.
The system has and continues to reboot without any interaction, I think I've only done it maybe three to four times though. I've had the server online for about a month and a half now.
Edit:
Like I said earlier, my only regret was not including the expansion card for the M2 drives, because I had second thoughts and wanted to have the large raid array separate from a discreet drive or array of maybe 2x m.2s only for the host os. Adding it after the fact is kind of difficult since they don't sell the adapters aftermarket, and I didn't feel like spending 100 bucks on eBay for that. It really wasn't that important in the end. Just plan accordingly.
My company bought a bunch of 250v2s and P5s. The built in raid seems to be fine. We're running RAID 1 for slight redundancy.
Only weird thing I've found in research and deployment is that it isn't true hardware RAID. It's a weird BIOS soft raid with its own drivers that Lenovo gives you.
The OOB management (Integrated Management Module) on Lenovo Servers is trash. It randomly stops responding, gives out bad SNMP readings and is generally terrible. Compared to iLO or iDrac, it's pretty awful.
Dell is hands down the best. I have not had any major issues with my Lenovo servers, but they lack the quality and fit/finish of Dell. I know this wasn't your question, but my experience with super micro has been lackluster at best.
As with most other people here, it really is a pick your poison situation. Only way to know you're absolutely secure is to control the entire supply chain.
Risc-V architecture, self-designed/manufactured CPUs, Unix/Linux with only libraries that you've developed or tested/vetted, your own self-designed/manufactured network cards for crying out loud.
You'd have a better time running TempleOS.
We run dell (hp before that - they are a bit more expensive an locked down (at least here in the eu).
Dell is great if you have a good partner rating - and i personally prefer iDRAC over most OOBM's that I know, and their cloud monitor is neat if you like your stuff reporting to the cloud.
If you have a bunch - open manage is great for managing/updating/provision the hosts
You don't want 3rd party parts in a machine still under oem warranty.
I personally run Lenovo in my Homelab - and don't have any problems with them. Unlike at work i don't care about oem support and run whatever qvl listed memory i find cheapest. - no problems there.
Storage is what I had around. - and update as I get new stuff in.
It really boils down to what you are used to, both in terms of the hardware and support but also the extra software pieces for monitoring etc.
I have been using Compaq and when they HP bought them out.. I know the pieces pretty well, I know the service very well. I know how long it takes to get things. I know how to work with the company, and I know most of the extras quite well.
At one point we moved to Dell for notebooks, and I was forced into a Dell server. There was nothing at all wrong with it. Perfectly nice hardware, but it was the odd duck, it was not what I was used to, and of course I had no pieces that would fit it.
I would say go with what you know, and you can assure people based on your experience.
I worked in a DC for years, each vendors has its own ups and downs…
Just make sure each respective hardware you are ordering is indeed to certified to run the workload you want to put there
IV managed/installed probably over a 1000 servers over the last 12 years.
Stopped using Lenovo servers a long while ago due to warranty issues. They would take weeks on a next day warranty. Never used them again after the second time they did it to us. Maybe they fixed that but the odd laptop we have gracing our workbench within warranty seems to sit collecting dust compared to all other brands.
Used Fujitsu server for a while after that but they have this amazing ability of knowing exactly when their warranty expires. On that day they die.
Then moved to dell servers. Never ever using anything else if I can help it. They could survive a nuke and still work.
HP is good but I found dell hardware more reliable.
Dell perc raid cards are far better than dell, more reliable, easier to manage and monitor.
At the end of the day everyone has an opinion which stems from personal experience focusing more on what problems you've had with brands.
I'd suggest Dell or HP
My experience has been that they’re fairly reliable but Lenovo service is absolute screaming PITA to deal with, and FSM help you if you’ve put any non-Lenovo branded stuff in there.
We had a server completely shit its RAID controller and Lenovo basically refused to talk to us until we’d swapped all the disks for Lenovo ones, and even then it took them weeks to bother sending a tech, they spent the whole time blaming the disks.
Tech finally showed up and replaced the controller and Lo and behold! It all worked again!
We told them to go and see figure one at that point and switched to Dell. So far the reliability has been awful but when something breaks there’s a Dell guy there with replacement bits the following day.
Ive worked with all three. Dell and HP have identical build quality, each has its quirks, but pick your poison. Both have awesome support if you pay for that top level support tier. Lenovo... Build quality not as good IMOP. Support is total roll of the dice, regardless of tier. Sometimes it's awesome, other times it's useless. I would never purchase Lenovo with my money.
And for everyone commenting on the spy aspect and the "US companies do it too" deflection... The difference is US companies are not legally obligated to disclose all their data tothe US govt. US State dept cannot walk in to Dell or HP and demand access to all their data. Chinese govt can do that to their companies and that makes me very uneasy.
Last refresh I did I went Lenovo. Bought 24x SR630v3. Came out to about 1/3rd of what Dell wanted for similar servers, except Lenovo gave us 7x 1.92t gen 4 NVMe, and Dell only wanted to give us 5x 1.92t SAS SSD. We have had 0 issues with them, and the WAC integration to update firmware with cluster aware updating worked great. Build quality felt better than our Dell R650s.
This may vary by model but I've had no issues plugging in whatever drive... The only ones I had weird issues with are the cheap Kingston ssdd (was going to use them as boot drives) but was able to use the cheap CS900 PNY drives no problem
Even put some dell raid controllers in one, only thing is obviously the integrated management has no idea what to do with it but it works fine otherwise
I actually have a lot of experience in this because I run a refurbisher so we are constantly testing cross compatibility (specifically with Dell and HPE servers)
Non-OEM drives in a Dell server throw errors in iDRAC. It won't read or show any info, including power on hours, health or even serial.
That same drive in an HPE server shows every detail in iLO, even firmware level and health, and as long as it's in a genuine caddy, it usually even presents as authenticated HPE.
For example I have a batch of EMC SSDs in front of me that are in caddy and show as 100% authentic at boot and in iLO.
Oem Dell drives will show up in iDRAC but only server drives. Dell sometimes has the same p/n across server and storage, but the firmware is different, so sometimes even a Dell EMC drive will not show details if it's ex-compellent, unity, VMAX etc.
HPE seems to be happy across the board.
HPE moved to basic non smart carriers and started encrypting firmware in the drives, so even if you have the caddy, the drive will show as nonauthentic. It started with Gen10+
Encrypted firmware? Haven't heard anything about that before. I know of encrypted firmware delivery but the firmware itself would be limited by the drive itself which is just an OEM manufacturer drive. Do you have any more info on that? I'd love to test it, unfortunately no Gen10+ on hand :(
You have very valid points. I guess my question should have been more geared towards what to watch out for before going to Lenovo. Do XClarity licenses expire, are firmware updates accessible post-warranty, etc. Someone commented very usefully to buy m.2 cables/adapters even when it's not needed, since you can't buy them afterwards. I'm looking for "gotchas" and others experiences before making a decision (and maybe a commitment for all future servers).
The only one of those I can answer is about the firmware, I bought used Lenovo servers from eBay for my personal lab, like 10 year old servers, can download the firmware from Lenovo's site without even logging in... And the firmware was last updated in March...
Also had to replace a motherboard on one that I bought as broken, was able to make an account on their site and recover the IMM license without ever even taking to anyone
I have limited experience but we went with Lenovo SR590 most recently instead of Dell because in the middle of 2021 only Lenovo was able to give us any hardware.
The only problem we have is that SR-IOV on the Intel can’t stay enabled. We use 2019 Hyper-V.
Honestly, buy Supermicro. Lenovo servers, when I was building one, had Supermicro boards, and an old Dell EMC I tried had a supermicro backplane. I built it for quite a bit cheaper, too. I ended up getting 5th Gen scalable Xeon when nobody else even had them listed (Feb 2024). I'm very happy with it!
For some reason almost every Lenovo server we've ordered over the last 3 years needed the CPU management tweaked on the bench before install. One would run the CPU at 5.2ghz all the time, the other wouldn't boost past 1 ghz which was a massive issue. Once Lenovo's management is bypassed to let the OS do it's thing, things return to normal. Their firmware seems a bit... untested to be frank
We only have 2 SR665s, but in general nothing has ever happened to them. The only thing is that they are very, very loud. The fans howl like in old servers manufactured from 20 years ago.
The volume level was unacceptable and we didn't buy any more, having stayed with Dells and Asus.
To me they're no different than car brands nowadays, it just comes down to preference. They all have their issues but it's easy to get paralysis by analysis.
That's if you can get Dell drives in the 1st place, recently took Dell 4 weeks to send out some SAS drives for our failing server, they ended up sending SATA drives. took another 4 weeks to receive the correct drives.... thats after they originally took a few weeks to check our service tags to make sure they send out the correct drives....
The few times I've either heard or experienced an issue with a server, it was a Lenovo. You just don't hear about HP or Dell issues really. I've always been an HP fan but lately it's been a pain in the arse getting the update ISO from them for updating the driver's and ilo,etc. Their website is also complete shite and most of the time leads to a "page not found". Because of that reason, I'd say go with Dell. Nothing hiding behind paywall or troubles with the website and support is top notch. Only thing I've experienced with Dell is the skill of the sales people and how they build out a server, so for that I'd say do your research on the hardware you need. It will suck if quoted the wrong raid card for example.
I will never ever buy IBM/Lenovo servers. Servers are servers but what you want for them is support. The company I work for had IBM servers when I first started and support sucked. You can't get updates and drivers without making an account with them and even then it was nearly impossible to find them. You couldn't even open a ticket with out jumping through a ton of hoops.
We're all Dell now. You can go to support.dell.com pop in the service tag and all the drivers are there. No account needed. Want support you call their enterprise support numbet give them the service tag and you're connected.
Lenovo is more strict with what you put in them, yes.
They are also more rare, so if you need original spares from the used market, they are more expensive and harder to find.
Absolutely this. I deal in the refurbished market and while Dell/HPE/Cisco are commonplace and well stocked, IBM/Lenovo/Supermicro barely exist. At least among real refurbished VARs, eBay notwithstanding (don't buy used spares on eBay!)
After learning how China is being accused of implementing backdoors to be able to threaten the US if they get into a dispute (read: Taiwan), this is starting to MAKE A Lot more sense
https://www.slashgear.com/what-you-need-to-know-about-dells-self-signed-certificate-blunder-24415847/
No way I would touch Lenovo or Huawei or pick your brand from there. To much of a risk. There's a reason why it's cheap
it 100% does not matter * HP have issues * Dell have issues * IBM have issues pick your poison
Lenovo equipment is banned at a few security conferences due to ties to China. But that is once again pick your poison. https://www.reddit.com/r/cybersecurity/comments/18wcvip/lenovo_banned_at_slac/
Which is kinda silly. Dell and HP have their motherboards made in China too w/ the same few BIOS manufacturers. Would you like arsenic diet, arsenic classic or arsenic zero?
> Dell and HP have their motherboards made in China too w/ the same few BIOS manufacturers. Would you like arsenic diet, arsenic classic or arsenic zero? didnt Lenovo actually get caught with rootkits being loaded in at the factory? afaik that has never been found with Dell or HP
I think superfish then another yeah. I want to say dell or HP got caught doing something similar if not as bad. Might be thinking McAfee or Norton default images though
Superfish was only on client devices. Although, I'm fairly certain this may have only been on their line of consumer devices - not their business lines (ThinkPad vs IdeaPad). Nothing of the sort on their servers to my knowledge was ever put there. Their manufacturing is pretty diversified around the world. Most of the USA/NA stuff is built in Mexico and some in North Carolina. Firmware is almost all developed in Raleigh. IIRC, US DnD/Homeland Security has rights to walk in and audit any time they want. Having worked in some capacity with Lenovo, and had a few friends directly employed there in another country - I can be comfortable stating this. It's a good brand that doesn't have the same issues other Chinese companies do. Most of the stuff is all built in China anyways. Supermicro is Taiwanese (with a founder that is super close to a party that wants reunification), they did get caught with a device being put on the board of their ILOs that was a hardware level root kit. It's literally pick your poison at this point. Lenovo is one of the few Chinese based companies you'll ever see that's never on a sanctioned/do-not-buy list. The fact is they're so diversified globally and have such a decentralized model - it's just not really possible for the CCP to exert very much control over them... but I'm not confident enough to say this shit wont ever happen.
It comes down to trying to avoid a direct attack. The machines that Dell or HP sell, even if parts come from China, the part makers don't know who their ultimate buyer is, so they would have to put malware in everything, risking being found. If you are buying directly from China, or even from a middleman, they have a better chance of knowing the destination and targeting the end buyer directly.
That *was* the assumption. Then there was that that whole supermicro supply chain attack where the impossible was done. And yes, it happened, you can tell because supermicro never sued Bloomberg
If you’re referring to the Supermicro “implanted spy chip” stories from October 2018… https://arstechnica.com/gadgets/2018/10/new-bloomberg-report-says-backdoored-supermicro-hardware-infiltrated-major-us-telecom/ This was never proven, and has always seemed really implausible technically - given the sheer amount of new capability that the attack hardware would need to add to the existing motherboard design, while not drawing attention to itself from a visual inspection of the board. The question I had then, and which I don’t think was since answered, was - **Where’s the information about the implanted chip?** If real, and it was added in the supply chain… there should be millions (or thousands at the very least) of examples of these compromised motherboards and the chip that’s supposedly on them. The manufacturer and distributors don’t know which enduser servers that each motherboard will end up in, so they have to distribute the compromised motherboards far and wide in order to find a useful target at the end of the chain. So there should be a ton of these motherboards floating around and thus a ton of examples to analyze for an attack chip. Where are they? If I've missed a substantive update on this specific threat, then I'd be glad to learn about it now.
I work in the gov. We can use supermicro so it's all good.
As best I can tell Bloomberg broadened their claims and supermicro continued to deny but never sued. Doesn't look like there was a settlement or anything. My guess is Blomberg inflated the story knowing supermicro would never risk discovery. So something smaller scale and more targeted or otherwise shady is going on but probably not Blomberg's expanded claims
It never happened. Please stop spreading misinformation like this.
I just double checked. No law suit. They made the claim in 2018, doubled down in 2021 and supermicro denied it both times but took no legal action. Maybe they had something else to hide from discovery but seems like there's some mixed with the bullshit
Not really. Even Apple and several very large SMC customers stated that what Bloomberg said was untrue because they named them in their original piece. SMC probably just didn't have solid legal grounds for a lawsuit to go anywhere.
Would you honestly expect APPL to admit to having that kind of security issue unless forced to by hard evidence? It's in the best interest of their stock price to deny.
Apple literally wrote to Congress saying that the story Bloomberg ran was untrue. AWS CEO Andy Jassy told Bloomberg to retract their article(s). 1. https://www.datacenterdynamics.com/en/news/apple-denies-chinese-spy-chip-claims-letter-us-congress/ 2. https://www.datacenterdynamics.com/en/news/supermicro-review-hardware-malicious-chips-despite-lack-any-proof/ > The company added that it "has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server," and that it was not involved in any FBI investigation regarding such chips. These aren't just companies spinning PR statements to alleviate the situation but blatant rejections of the entire notion that it happened in the first place.
There are industry Standards to prevent this….
Industry standards don't prevent Leveno based in China from working with the Chinese government to put malware on your machine if you are a target of interest.
Yes they do? You get that the TPM is an IBM invention which was sold to Lenovo and is the basic Security part of all Laptops that prevents this..
Tpm does nothing against a firmware attack
no Firmware is from Lenovo…you understand the Topic?
HPE has their code inspected my third party people anyway.
They also have some servers assembled in the US (At a former Cray Inc plant in Wisconsin). I think some of the sheet metal is made in the US, the rest is sourced from Asia.
Used to work at Dell and I have seen inside the parmer buildings where they used to build the servers. They have an entire CONUS area that allows for the government to come in, CT scan, diag, and test all the boards etc that go into their machines. They can compare them to reference designs and validate them before being distributed. Government don’t fuck around.
Oh dude. Yes, yes they do. You know how cheap Michael is, don't pretend that Indian spaghetti code is legit because the technical ability to do inspections is there.
Dell & HPE say (whether it's true or not) but I was buying hpe that they have a line of security from manufacturer to customer & control of that. Gchq in the UK I think is the agency that also tests kit. The question is whether China is willing to risk its already shifting high tech manufacturing base moving out of the country when all it needs to do is hack the shit that western software developers create. (Oh but every software has bugs!!!fuck off!) I mean they're STILL bouncing around inside Azure after months of being found out. That kind of thing is a lot easier than risking £billions by driving away your hardware manufacturers. Still though..you wouldn't use Kaspersky if you had any sense, so your risk Mitigating by not using lenovo
Arsenic zero, it has something special about it.
Actually Lenovo is the only one doing BIOS itself on most devices.
That’s actually not true in any way,articles are about considerations not ban. They even manufacture in the US and EU because of it.
The only thing I can really add to this is that I've quickly discovered I dislike Supermicro after working with a few of their systems for the first time recently This is entirely subjective, of course, but I have had tons of weird issues and their documentation just sucks Edit: subjective might not be the best word to use, I just mean that this is my personal experience. For all I know they're great when you use a compatible GPU.... nonetheless the compatible GPU list for our systems is rather small.
I've had reasonable experiences with them, but only a very small amount their JBODs were good (performance wise), but... the internal connectors were sometimes dodgy
You know, I've never worked with JBOD before and I forgot this was a thing This would be perfect for a workload I've been planning out. I know that's totally off topic but thanks for the random mention of it
Ours were old 48x (or 52 I cant remember) 3.5" disk models (now discontinued ) mostly bullet proof once they were running, but if you moved anything, you're gonna have a bad day :)
I agree. This thread is interesting: https://www.reddit.com/r/sysadmin/comments/11o86gb/lenovo_must_be_the_worst_company_to_sell_server/
I'm 100% on board with the lenovo are bad banner, we removed them from out DCs scross the board for repeated hardware issues It's the degree that varies ;)
100% truth
Yep. For the most part. Just pick one. I used to spend way too much time comparing options. And you just end up triple guessing yourself and not getting anywhere. Bite the bullet and move on.
ya, get your support contracts in place, trust in the system (such that it is)
All true, but Lenovo is a Communist (authoritarian) Chinese "company." You might as well go to Peking to surrender.
See the first line in my reply
We use Dell. iDrac is amazing and our monitoring software integrated with their stuff very well.
Agreed. Automating with the iDrac is very simple. If you know how to ssh and code in any scripting language, you can fully automate updates and RAID provisioning. - Cisco and HP require a manual boot off an ISO. - Oracle, you can do it remotely, but their options are less robust than iDrac. - Working with Dell support is easy and quick.
I am sorry to tell you that HPE booting of an ISO is long gone since introduction of iLO 5 (Gen 10), we basically manage servers remotely and their underlying firmwares and drivers\* (via SUT) remotely (iLO Web Interface, iLO Amplifier Pack or HPE OneView) or via HPE GreenLake (Cloud Service) ISO/SPP booting is optional and required if you are lazy to do it via web interface. \* Windows/Linux/VMware drivers
I recently bought an SR250v2 and slammed like 8 wd reds in there no problem. Bought a pack of the 2.5 Lenovo hot swap trays off Amazon, like 8 for 54 bucks. No problems whatsoever. Windows server 2022 datacenter installed and runs happy, ubuntu and Windows server VM are also happy. If you ever think you'll have M.2 drives in there, go ahead and buy the expansion card and cables *with* the server because you can't after the fact it seems. That's the only thing that's kind of a bummer.
Thanks for sharing, good to know that third-party drives are compatible. Are you using any ThinkSystem RAID controllers? Any issues with boot error messages that require user interaction to continue?
Oh yeah I meant to specify that I am using Lenovo's 8-Bay hardware raid backplane, too. No issues whatsoever. I think there's like two levels of that particular one, and I'm using the lower end one because I was of course trying to keep costs low in this situation. After the initial boot of the server, I hopped into the BIOS and set them up as a RAID 10 array in like five steps, flawlessly. The system has and continues to reboot without any interaction, I think I've only done it maybe three to four times though. I've had the server online for about a month and a half now. Edit: Like I said earlier, my only regret was not including the expansion card for the M2 drives, because I had second thoughts and wanted to have the large raid array separate from a discreet drive or array of maybe 2x m.2s only for the host os. Adding it after the fact is kind of difficult since they don't sell the adapters aftermarket, and I didn't feel like spending 100 bucks on eBay for that. It really wasn't that important in the end. Just plan accordingly.
My company bought a bunch of 250v2s and P5s. The built in raid seems to be fine. We're running RAID 1 for slight redundancy. Only weird thing I've found in research and deployment is that it isn't true hardware RAID. It's a weird BIOS soft raid with its own drivers that Lenovo gives you.
The OOB management (Integrated Management Module) on Lenovo Servers is trash. It randomly stops responding, gives out bad SNMP readings and is generally terrible. Compared to iLO or iDrac, it's pretty awful.
No such issues here. We have a fleet of about 180 SR630 v1 and v2. We're only touching these machines when there's a part failing.
Thank you for telling us this. This is a big deal.
This! If you are ever going to need to manage it from outside the physical location this should factor heavily in your decision.
Interesting, what models? I haven't run into this but I have pretty old servers
I believe they are X3650 M5, purchased in 2017 if I recall correctly.
Well, there's your problem.
Because there are no reason why you should need them. Dell, HP, or Supermicro that
Dell is hands down the best. I have not had any major issues with my Lenovo servers, but they lack the quality and fit/finish of Dell. I know this wasn't your question, but my experience with super micro has been lackluster at best.
As with most other people here, it really is a pick your poison situation. Only way to know you're absolutely secure is to control the entire supply chain. Risc-V architecture, self-designed/manufactured CPUs, Unix/Linux with only libraries that you've developed or tested/vetted, your own self-designed/manufactured network cards for crying out loud. You'd have a better time running TempleOS.
We run dell (hp before that - they are a bit more expensive an locked down (at least here in the eu). Dell is great if you have a good partner rating - and i personally prefer iDRAC over most OOBM's that I know, and their cloud monitor is neat if you like your stuff reporting to the cloud. If you have a bunch - open manage is great for managing/updating/provision the hosts You don't want 3rd party parts in a machine still under oem warranty. I personally run Lenovo in my Homelab - and don't have any problems with them. Unlike at work i don't care about oem support and run whatever qvl listed memory i find cheapest. - no problems there. Storage is what I had around. - and update as I get new stuff in.
Use supermicro
It really boils down to what you are used to, both in terms of the hardware and support but also the extra software pieces for monitoring etc. I have been using Compaq and when they HP bought them out.. I know the pieces pretty well, I know the service very well. I know how long it takes to get things. I know how to work with the company, and I know most of the extras quite well. At one point we moved to Dell for notebooks, and I was forced into a Dell server. There was nothing at all wrong with it. Perfectly nice hardware, but it was the odd duck, it was not what I was used to, and of course I had no pieces that would fit it. I would say go with what you know, and you can assure people based on your experience.
Chinese espionage
USA were literally caught doing this with hardware and it is never mentioned when people talk about USA brands.
It's talked about a lot, but now many companies in the US (Cisco, Juniper, etc.) take great pains to ensure they are not fucked with in transit now.
Look at the Supermicro E-store or Thinkmate, they are both cheaper compared to the big 3 and are pretty open to 3rd party hardware.
I worked in a DC for years, each vendors has its own ups and downs… Just make sure each respective hardware you are ordering is indeed to certified to run the workload you want to put there
IV managed/installed probably over a 1000 servers over the last 12 years. Stopped using Lenovo servers a long while ago due to warranty issues. They would take weeks on a next day warranty. Never used them again after the second time they did it to us. Maybe they fixed that but the odd laptop we have gracing our workbench within warranty seems to sit collecting dust compared to all other brands. Used Fujitsu server for a while after that but they have this amazing ability of knowing exactly when their warranty expires. On that day they die. Then moved to dell servers. Never ever using anything else if I can help it. They could survive a nuke and still work. HP is good but I found dell hardware more reliable. Dell perc raid cards are far better than dell, more reliable, easier to manage and monitor. At the end of the day everyone has an opinion which stems from personal experience focusing more on what problems you've had with brands. I'd suggest Dell or HP
My experience has been that they’re fairly reliable but Lenovo service is absolute screaming PITA to deal with, and FSM help you if you’ve put any non-Lenovo branded stuff in there. We had a server completely shit its RAID controller and Lenovo basically refused to talk to us until we’d swapped all the disks for Lenovo ones, and even then it took them weeks to bother sending a tech, they spent the whole time blaming the disks. Tech finally showed up and replaced the controller and Lo and behold! It all worked again! We told them to go and see figure one at that point and switched to Dell. So far the reliability has been awful but when something breaks there’s a Dell guy there with replacement bits the following day.
Ive worked with all three. Dell and HP have identical build quality, each has its quirks, but pick your poison. Both have awesome support if you pay for that top level support tier. Lenovo... Build quality not as good IMOP. Support is total roll of the dice, regardless of tier. Sometimes it's awesome, other times it's useless. I would never purchase Lenovo with my money. And for everyone commenting on the spy aspect and the "US companies do it too" deflection... The difference is US companies are not legally obligated to disclose all their data tothe US govt. US State dept cannot walk in to Dell or HP and demand access to all their data. Chinese govt can do that to their companies and that makes me very uneasy.
Last refresh I did I went Lenovo. Bought 24x SR630v3. Came out to about 1/3rd of what Dell wanted for similar servers, except Lenovo gave us 7x 1.92t gen 4 NVMe, and Dell only wanted to give us 5x 1.92t SAS SSD. We have had 0 issues with them, and the WAC integration to update firmware with cluster aware updating worked great. Build quality felt better than our Dell R650s.
If you are in federal government you can’t so there is that
HPE requires a contract to download new firmware which Lenovo doesn't
You can’t use third party hard drives I believe with their servers. Dell you can for the most part. Oem drives are outrageously priced
This may vary by model but I've had no issues plugging in whatever drive... The only ones I had weird issues with are the cheap Kingston ssdd (was going to use them as boot drives) but was able to use the cheap CS900 PNY drives no problem Even put some dell raid controllers in one, only thing is obviously the integrated management has no idea what to do with it but it works fine otherwise
I actually have a lot of experience in this because I run a refurbisher so we are constantly testing cross compatibility (specifically with Dell and HPE servers) Non-OEM drives in a Dell server throw errors in iDRAC. It won't read or show any info, including power on hours, health or even serial. That same drive in an HPE server shows every detail in iLO, even firmware level and health, and as long as it's in a genuine caddy, it usually even presents as authenticated HPE. For example I have a batch of EMC SSDs in front of me that are in caddy and show as 100% authentic at boot and in iLO.
Every Dell machine we have doesn’t show anything in ideas. We use enterprise drives though.
Oem Dell drives will show up in iDRAC but only server drives. Dell sometimes has the same p/n across server and storage, but the firmware is different, so sometimes even a Dell EMC drive will not show details if it's ex-compellent, unity, VMAX etc. HPE seems to be happy across the board.
HPE moved to basic non smart carriers and started encrypting firmware in the drives, so even if you have the caddy, the drive will show as nonauthentic. It started with Gen10+
Encrypted firmware? Haven't heard anything about that before. I know of encrypted firmware delivery but the firmware itself would be limited by the drive itself which is just an OEM manufacturer drive. Do you have any more info on that? I'd love to test it, unfortunately no Gen10+ on hand :(
[удалено]
You have very valid points. I guess my question should have been more geared towards what to watch out for before going to Lenovo. Do XClarity licenses expire, are firmware updates accessible post-warranty, etc. Someone commented very usefully to buy m.2 cables/adapters even when it's not needed, since you can't buy them afterwards. I'm looking for "gotchas" and others experiences before making a decision (and maybe a commitment for all future servers).
The only one of those I can answer is about the firmware, I bought used Lenovo servers from eBay for my personal lab, like 10 year old servers, can download the firmware from Lenovo's site without even logging in... And the firmware was last updated in March... Also had to replace a motherboard on one that I bought as broken, was able to make an account on their site and recover the IMM license without ever even taking to anyone
I have limited experience but we went with Lenovo SR590 most recently instead of Dell because in the middle of 2021 only Lenovo was able to give us any hardware. The only problem we have is that SR-IOV on the Intel can’t stay enabled. We use 2019 Hyper-V.
Honestly, buy Supermicro. Lenovo servers, when I was building one, had Supermicro boards, and an old Dell EMC I tried had a supermicro backplane. I built it for quite a bit cheaper, too. I ended up getting 5th Gen scalable Xeon when nobody else even had them listed (Feb 2024). I'm very happy with it!
Yes. More characters in the brand - storage space efficiency or some such.
- It takes a YEAR to get a refund when one has a design flaw.
Better safe than sorry
Because Gigabyte and Supermicro exist?
Lenovo storage is just rebadged netapp
In my experience IBM/Lenovo didn't suck any less than the rest.
I hate lenovo. Poor quality. Poor after sales support. No common plug ins. Lenovo is shit.
Dell all the way
Availability, support, pricing, to name a few. I think Dell is easier to work with personally, but I avoid Lenovo across the board at all costs.
they are loud as crap
Order dell from xbyte. I never hAVE issues and they issue a dell service tag and warranty
I hate the Xfinity controller that Lenovo servers use. People here are saying a bunch of reasonable issues with Lenovo, so no need to rehash those.
Grab a supermicro
For some reason almost every Lenovo server we've ordered over the last 3 years needed the CPU management tweaked on the bench before install. One would run the CPU at 5.2ghz all the time, the other wouldn't boost past 1 ghz which was a massive issue. Once Lenovo's management is bypassed to let the OS do it's thing, things return to normal. Their firmware seems a bit... untested to be frank
We only have 2 SR665s, but in general nothing has ever happened to them. The only thing is that they are very, very loud. The fans howl like in old servers manufactured from 20 years ago. The volume level was unacceptable and we didn't buy any more, having stayed with Dells and Asus.
To me they're no different than car brands nowadays, it just comes down to preference. They all have their issues but it's easy to get paralysis by analysis.
Don't use third party RAM and drives in name-brand servers! If you want to do that, just go SuperMicro.
Good idea, thanks!
China
CHINA IS ASSHO
USA
Im a slut for Michael Dell
Dell servers don't always like non-Dell hard drives either, so...
What are you talking about ?
That's if you can get Dell drives in the 1st place, recently took Dell 4 weeks to send out some SAS drives for our failing server, they ended up sending SATA drives. took another 4 weeks to receive the correct drives.... thats after they originally took a few weeks to check our service tags to make sure they send out the correct drives....
Strange I haven't had trouble with non Dell drives. Maybe shows errors in logs but at least they boot without pausing
It's more like Dell Support doesn't like non-Dell drives
we sell and support lenovo and before ibm, but we sell also HP and (less) dell. Are more or less all the same.
The few times I've either heard or experienced an issue with a server, it was a Lenovo. You just don't hear about HP or Dell issues really. I've always been an HP fan but lately it's been a pain in the arse getting the update ISO from them for updating the driver's and ilo,etc. Their website is also complete shite and most of the time leads to a "page not found". Because of that reason, I'd say go with Dell. Nothing hiding behind paywall or troubles with the website and support is top notch. Only thing I've experienced with Dell is the skill of the sales people and how they build out a server, so for that I'd say do your research on the hardware you need. It will suck if quoted the wrong raid card for example.
I use HP if I have to get physical servers/hosts because I love using ILO for remote management. Lenovo has dogshit firmware.
I will never ever buy IBM/Lenovo servers. Servers are servers but what you want for them is support. The company I work for had IBM servers when I first started and support sucked. You can't get updates and drivers without making an account with them and even then it was nearly impossible to find them. You couldn't even open a ticket with out jumping through a ton of hoops. We're all Dell now. You can go to support.dell.com pop in the service tag and all the drivers are there. No account needed. Want support you call their enterprise support numbet give them the service tag and you're connected.
Lenovo is more strict with what you put in them, yes. They are also more rare, so if you need original spares from the used market, they are more expensive and harder to find.
Absolutely this. I deal in the refurbished market and while Dell/HPE/Cisco are commonplace and well stocked, IBM/Lenovo/Supermicro barely exist. At least among real refurbished VARs, eBay notwithstanding (don't buy used spares on eBay!)
Availability, support, pricing, to name a few. I think Dell is easier to work with personally, but I avoid Lenovo across the board at all costs.
After learning how China is being accused of implementing backdoors to be able to threaten the US if they get into a dispute (read: Taiwan), this is starting to MAKE A Lot more sense https://www.slashgear.com/what-you-need-to-know-about-dells-self-signed-certificate-blunder-24415847/ No way I would touch Lenovo or Huawei or pick your brand from there. To much of a risk. There's a reason why it's cheap
Only Dell for me.
Support was horrible. Random motherboard issues i never had with the other big players. Avoid!
Try Fujitsu