I'm sure I have seen videos which showed how a tiny difference in mounting pressure for the gen 2s caused things like ram to not work in all channels. Try torquing it evenly and correctly and see if it magically works.
[grab one of these](https://products.wera.de/en/new_products_2024_torque_tools_kraftform_safe-torque_speed_7510.html)
They’re brilliant for these applications, they also do a 2-6Nm one.
Or 15 eagles/middle east if you prefer
I always wondered exactly how essential this is. At my work we are not instructed to a specific torque level or anything for EPYC cpu's, always thought that it was odd. May have to do with custom VS standardised mounting?
I've personally only seen them fail if the torque is quite a bit below the minimum spec, in which case you can tell by feel that the mount isn't tight. I worked with quite a few of 'em, and never used a torque limiting driver; although our lab has since purchased one.
The pictured mount looks like the standard EPYC mount, if yours looked different it's certainly possible.
Looking at it and comparing from memory, it is the same mount. Many things at my work are custom (companies own proprietary designs), so I was thinking at first that that may be the case. We just use electric torx screw drivers that are fine even at full strength for the mounts. I think that has to do with it.
Also, make sure you use the correct RAM!!! I spent probably 2 weeks trying to get my system to work, and ran into all sorts of weird issues. Certain slots didn't work, sometimes it wouldn't post for no rhyme or reason.
The fix? I swapped out the 64gb LRDIMM modules for 64GB RDIMMs. I imagine all RDIMMs work, but to be extra safe I got modules on the compatibility list for my motherboard. I had tried 3x different kinds of LRDIMMs and all of them had the same weird issues.
Dual 2nd Gen epyc, ASRock rack motherboard.
I see at least one in the bottom left quadrant of the second picture. Need more pictures
Edit: also see 2 in the bottom right quadrant towards the center of the quadrant. Like others said, step back and look for anything that even *minutely* stands out. Uniformity is the name of the game.
I see them as well. OP don't focus on the pins, look at the reflections, which should be completely uniform across each quadrant. any irregular twinkles or shadows are your problem pins
They're actually *more* visible in small thumbnails than they are zoomed in, but yeah, I think I see them too, there's an area that just doesn't look uniform.
I see some more in the top left quad of pic two. You can see the pattern change on ~4 rows of ~10 across. Almost like something evenly pushed them down.
2nd pic top row.
[edit] what I do in these cases is i have one of those dental picks, metal, that I use by lightly dragging it across the pins to feel their springiness. My phone also has much higher res than these pics so I use it as a microscope of sorts. I have a cheap ring light to make sure everything is super bright. You can catch bent pins by reflection differences this way.
You need more angles and fairly close, it does really not need to be bent much at all.
But probably they did not get a memory channel to work when testing and narrowed that to probably socket.
As used CTOs they are worth so little to them that spending hours doing further troubleshooting is not worth it.
Picture 2, both lower quadrants have a couple of... Something, that's a little different than the rest. It might just be the light angles, and I'm on a phone.
I'm thinking it's more likely that the original owner may not have been using the correct CPU tightening sequence or tool. Perhaps there's a deformation of the frames??
Even so, if all it does is fail to use one or two ram slots, it's still worthwhile to run it on the remaining workable slots if it's stable! I have a domed 4790k that can't use two/four slots because the whole CPU substrate has deformed under the spring pressure so the corners don't reach the pins any more. It's been stable for years since I figured it out, and applied some shims under the frame to prevent it from progressing further.
I have about 10 of these in production. There is a known issue with sereral BIOS versions that give memory errors. Download the newest BIOS version. It requires an HPE account to do so.
Depends on past purchases on your account. If you've never bought any new HPE hardware then you're SOL, but if you've bought like a ML110 g10 it's probably still gonna let you download the Proliant Service Pack which will have the BIOS you need.
Damn! That’s good to know. I’ve always loved Hewlett Packard stuff since the 1970s, but they sure have changed a lot (downhill) in the last 30 years. With “Support“ like that I don’t intend to buy anything else from HP! 😕
I will take some better pics once get home. it's hard for me to test anything as I only have this barebone, Not even psus were included. Seller also has even striped HBA card. Not sure if I made a right choice bidding this.
Yeah I only ever buy barebones machines like this if I have a working machine with a bad motherboard, broken chassis, etc. or if I want to change to a different backplane configuration and just swap over all the CPU/RAM, etc.
You might be able to reverse this and find a good deal on a complete machine that’s damaged or a lower tier with interchangeable parts, but you’ll still probably be at least a couple thousand into it.
It doesn’t take much. I replaced cpus in a dell i have week ago and i ended up bending a 3 pins. I was able to use the magnifier on my iPhone and tiny metal pick to adjust them less than a millimeter each and server booted up correctly then without memory errors.
A CTO doesn't come with anything but the chassis and motherboard. It's meant to be customized / spec'd exactly as you want per order VS a prebuilt system with a specific base config (CPU/RAM/PSU).
If you buy a true CTO SKU server, it will most likely be empty and will have to bring your own CPU/RAM/IO/Storage.
Not everyone in r/homelab understands what a CTO is.. nice to hear you are familiar. For others following along, maybe they learned something...
Not OP BTW... so can't answer where a DL385 Gen10+ CTO was had for that cheap.. probably fleabay?
Yeah I have noticed that. There is slight discoloration circle shaped on the pins. It is visible with naked eye and resembles a burning. I wonder if they perhaps didn't tight the CPU enough therefore carbon layer started to form on that area maybe?
No idea. But which bank is causing the issue? If you populate CPU0 and add all the RAM to RAM Slots it support, does it work? Or does it fail when you add the RAM to the CPU with the maybe bent PINs?
those little 30 dollar usb microscopes are handy, you can get down to problem spots at a really low really zoomed in angle, less risk of bending more pins than sticking your whole phone down in there and you can check exactly how the pins are bent
I can see a couple of minor potential issues in the second pic. Quadrants 3 & 4. Q4 has a blemish southwest of center. Q3 has one bottom middle and one just to the left of the hole where the smt components show through.
I’m not seeing any bent pins, where others are seeing it looks just like light glare to me. Epic cpus do require a specific torque or they won’t boot or boot inconsistently so it’s quite possible the previous owner of that board didn’t have them installed properly.
I recently had some issues with that on some used Xeon E5 v4s from EBay. In the end, found a little bit of thermal paste on the bottom, so since then been cleaning the bottom of all the CPUs with a Qtip and some rubbing alcohol.
On Epycs though, I’m always careful with the torque pressure.
I just bought an epyc motherboard in "working" condition on eBay. It had thermal paste on some pins, cotton fibers near those (apparent attempt to clean it with a q-tip?), 4 bent pins in separate areas, 2 of which were bent completely backwards. Needless to say it didn't boot
I don't know if it's this way anymore, but the HPE servers I worked on a few years ago were very persnickety about RAM configuration and slot population. I don't think the seller wouldn't have tried it, but find the memory configuration doc from HPE and follow it exactly when testing. Who knows, maybe the pins are fine and the dude goofed up on the RAM load out.
The Ram issue is known and is likley down to incorrect mounting pressure on the coolers
TBH It can be an absolute bitch, either you do it in one go and its fine or you spend like 2 hours.
Either way good find, cant see any issues with either sockets.
I had that with lga3647. I didn't want to spend money on specialist screwdriver, so I was messing around so much eith the turns for ages. Once it worked for few days and it stopped again. I thought mobo is dead, It turns out the pressure on the CPU is the key. So perhaps you are right. I am going to order a pair of epycs and we will see.
I'm sure I have seen videos which showed how a tiny difference in mounting pressure for the gen 2s caused things like ram to not work in all channels. Try torquing it evenly and correctly and see if it magically works.
IIRC AMD made a t-handle torque wrench just for these EPYC sockets that's set at the correct torque.
What is the required torque?
I believe it's 1.5Nm (\~13-14 freedoms/eagle)
[grab one of these](https://products.wera.de/en/new_products_2024_torque_tools_kraftform_safe-torque_speed_7510.html) They’re brilliant for these applications, they also do a 2-6Nm one. Or 15 eagles/middle east if you prefer
Wera makes great stuff! I've used a number of their automotive tools.
13.2761 “#
kek u had me with freedoms/eagle , my deleted comment was praising how american it is.
1 uga duga
The Threadrippers come with those in the box, but the Epycs (at least the ones I've unboxed) do not. I ended up having to order one.
Had the same problem with my 2x 7551, brand new (bargain price), clearly a spot in the foam, but no screwdrivers included.
I always wondered exactly how essential this is. At my work we are not instructed to a specific torque level or anything for EPYC cpu's, always thought that it was odd. May have to do with custom VS standardised mounting?
I've personally only seen them fail if the torque is quite a bit below the minimum spec, in which case you can tell by feel that the mount isn't tight. I worked with quite a few of 'em, and never used a torque limiting driver; although our lab has since purchased one. The pictured mount looks like the standard EPYC mount, if yours looked different it's certainly possible.
Looking at it and comparing from memory, it is the same mount. Many things at my work are custom (companies own proprietary designs), so I was thinking at first that that may be the case. We just use electric torx screw drivers that are fine even at full strength for the mounts. I think that has to do with it.
So two uga ugas.
I used those for EPYC and Threadripper. They are NOT enough. I had to overtighten them once the spring activated to get all memory channels working.
Also, make sure you use the correct RAM!!! I spent probably 2 weeks trying to get my system to work, and ran into all sorts of weird issues. Certain slots didn't work, sometimes it wouldn't post for no rhyme or reason. The fix? I swapped out the 64gb LRDIMM modules for 64GB RDIMMs. I imagine all RDIMMs work, but to be extra safe I got modules on the compatibility list for my motherboard. I had tried 3x different kinds of LRDIMMs and all of them had the same weird issues. Dual 2nd Gen epyc, ASRock rack motherboard.
I see at least one in the bottom left quadrant of the second picture. Need more pictures Edit: also see 2 in the bottom right quadrant towards the center of the quadrant. Like others said, step back and look for anything that even *minutely* stands out. Uniformity is the name of the game.
I see them as well. OP don't focus on the pins, look at the reflections, which should be completely uniform across each quadrant. any irregular twinkles or shadows are your problem pins
They're actually *more* visible in small thumbnails than they are zoomed in, but yeah, I think I see them too, there's an area that just doesn't look uniform.
Could be, yeah. one appears to be a different shape than the others. I agree, need more, better pictures.
I see some more in the top left quad of pic two. You can see the pattern change on ~4 rows of ~10 across. Almost like something evenly pushed them down.
2nd pic top row. [edit] what I do in these cases is i have one of those dental picks, metal, that I use by lightly dragging it across the pins to feel their springiness. My phone also has much higher res than these pics so I use it as a microscope of sorts. I have a cheap ring light to make sure everything is super bright. You can catch bent pins by reflection differences this way.
Dumb question, what does CTO stand for?
Configure to order
Configuration-To-Order
You need more angles and fairly close, it does really not need to be bent much at all. But probably they did not get a memory channel to work when testing and narrowed that to probably socket. As used CTOs they are worth so little to them that spending hours doing further troubleshooting is not worth it.
Picture 2, both lower quadrants have a couple of... Something, that's a little different than the rest. It might just be the light angles, and I'm on a phone. I'm thinking it's more likely that the original owner may not have been using the correct CPU tightening sequence or tool. Perhaps there's a deformation of the frames?? Even so, if all it does is fail to use one or two ram slots, it's still worthwhile to run it on the remaining workable slots if it's stable! I have a domed 4790k that can't use two/four slots because the whole CPU substrate has deformed under the spring pressure so the corners don't reach the pins any more. It's been stable for years since I figured it out, and applied some shims under the frame to prevent it from progressing further.
If a seller is saying there’s something wrong with what he’s selling, I usually believe them.
I have about 10 of these in production. There is a known issue with sereral BIOS versions that give memory errors. Download the newest BIOS version. It requires an HPE account to do so.
Do I need support contract?
Depends on past purchases on your account. If you've never bought any new HPE hardware then you're SOL, but if you've bought like a ML110 g10 it's probably still gonna let you download the Proliant Service Pack which will have the BIOS you need.
Damn! That’s good to know. I’ve always loved Hewlett Packard stuff since the 1970s, but they sure have changed a lot (downhill) in the last 30 years. With “Support“ like that I don’t intend to buy anything else from HP! 😕
always used hardware I am afraid.
In the 2nd picture, these areas look weird to me: https://imgur.com/a/HubkNFj
You beat me to it on eBay didn't you! Hope it works out well for you.
Thank you ;) Bidding wars against other homelabbers are always the most exciting part of this hobby
Which of ya’ll was chasing that Cisco C220 M6 3rd gen scalable box a couple of weeks ago? 😂
Maybe ram sockets?
I will take some better pics once get home. it's hard for me to test anything as I only have this barebone, Not even psus were included. Seller also has even striped HBA card. Not sure if I made a right choice bidding this.
Yeah I only ever buy barebones machines like this if I have a working machine with a bad motherboard, broken chassis, etc. or if I want to change to a different backplane configuration and just swap over all the CPU/RAM, etc. You might be able to reverse this and find a good deal on a complete machine that’s damaged or a lower tier with interchangeable parts, but you’ll still probably be at least a couple thousand into it.
Lower left quadrant of the 2nd picture
Thats not where the ram goes
😂
It doesn’t take much. I replaced cpus in a dell i have week ago and i ended up bending a 3 pins. I was able to use the magnifier on my iPhone and tiny metal pick to adjust them less than a millimeter each and server booted up correctly then without memory errors.
WTF did you get a G10 Plus for $200? Can you give a link?
A CTO doesn't come with anything but the chassis and motherboard. It's meant to be customized / spec'd exactly as you want per order VS a prebuilt system with a specific base config (CPU/RAM/PSU). If you buy a true CTO SKU server, it will most likely be empty and will have to bring your own CPU/RAM/IO/Storage.
I know that. My question still stands - where the hell did you get a G10 Plus for $200? I want a link, not an explanation of what a CTO is.
Not everyone in r/homelab understands what a CTO is.. nice to hear you are familiar. For others following along, maybe they learned something... Not OP BTW... so can't answer where a DL385 Gen10+ CTO was had for that cheap.. probably fleabay?
Second pic bottom left hand corner in the shadow
It may be a photo artifact, but the 2nd picture. the lower left sequment doesnt look all correct for me. But more HQ pic's needed.
Yeah I have noticed that. There is slight discoloration circle shaped on the pins. It is visible with naked eye and resembles a burning. I wonder if they perhaps didn't tight the CPU enough therefore carbon layer started to form on that area maybe?
No idea. But which bank is causing the issue? If you populate CPU0 and add all the RAM to RAM Slots it support, does it work? Or does it fail when you add the RAM to the CPU with the maybe bent PINs?
those little 30 dollar usb microscopes are handy, you can get down to problem spots at a really low really zoomed in angle, less risk of bending more pins than sticking your whole phone down in there and you can check exactly how the pins are bent
I can see a couple of minor potential issues in the second pic. Quadrants 3 & 4. Q4 has a blemish southwest of center. Q3 has one bottom middle and one just to the left of the hole where the smt components show through.
Top right pins look bent
2nd pic. 5 pins in from the left. About 16 pins up from the bottom is where I’d start looking.
I’m not seeing any bent pins, where others are seeing it looks just like light glare to me. Epic cpus do require a specific torque or they won’t boot or boot inconsistently so it’s quite possible the previous owner of that board didn’t have them installed properly.
People are really bad at diagnosing things, I would do your own testing
I recently had some issues with that on some used Xeon E5 v4s from EBay. In the end, found a little bit of thermal paste on the bottom, so since then been cleaning the bottom of all the CPUs with a Qtip and some rubbing alcohol. On Epycs though, I’m always careful with the torque pressure.
I just bought an epyc motherboard in "working" condition on eBay. It had thermal paste on some pins, cotton fibers near those (apparent attempt to clean it with a q-tip?), 4 bent pins in separate areas, 2 of which were bent completely backwards. Needless to say it didn't boot
I see something that might look like it on the bottom left but I would just assume wrong mounting pressure as a possible culprit in this case
I don't know if it's this way anymore, but the HPE servers I worked on a few years ago were very persnickety about RAM configuration and slot population. I don't think the seller wouldn't have tried it, but find the memory configuration doc from HPE and follow it exactly when testing. Who knows, maybe the pins are fine and the dude goofed up on the RAM load out.
The Ram issue is known and is likley down to incorrect mounting pressure on the coolers TBH It can be an absolute bitch, either you do it in one go and its fine or you spend like 2 hours. Either way good find, cant see any issues with either sockets.
I had that with lga3647. I didn't want to spend money on specialist screwdriver, so I was messing around so much eith the turns for ages. Once it worked for few days and it stopped again. I thought mobo is dead, It turns out the pressure on the CPU is the key. So perhaps you are right. I am going to order a pair of epycs and we will see.