pixy (DFI ITOX ST102-CS-SVR)

I was looking for something to replace Yamato since it is GPU crashy. Intel 8th gen seemed like a decent compromise between cost, performance, and something that I thought would be stable. I ran across this little server with two NICs in a compact ITX case. The price seemed OK, though it was more than other capable mini PCs. I hoped to avoid the whiny fans though so I went for it.

Auction Photo - Machine + PSU

Auction Photo - Front

Auction Photo - Rear Ports

Auction Photo - BIOS

I got the machine and opened it up. It had a 1TB 2.5" drive. I thought this was the Windows drive so I pulled it, but Windows kept booting. I looked around, but none of the M.2, Mini PCIe, or PCIe slots were occupied. Pulling the board I found another M.2 slot on the bottom populated with a 1TB NVMe SSD. It's a bit older so the write speed isn't all that, but 2.5 GB​/s reads, and 570 MB​/s writes is pretty good.

I didn't know what drives there were going in, but after finding these two I feel pretty good about how much I paid.

Drive tray with 2.5" drive

Machine with drive tray removed

Back of motherboard

1TB NVMe Micron SSD close-up

It turned out to be pretty loud. Due to the fan configuration only the CPU fan could be controlled. I happened to have a spare Thermalright AXP90 X36 cooler which fit the 75x75mm mount of the LGA 1151 socket. I installed it, but due to component locations I had to flip the mounting bracket. With the mounting bracket flipped it no longer fit behind the motherboard. So I modded the case.

The new heatsink is more massive, but the fan is very close to the drive mounting plate. I'm probably going to remove the plate, but I sketched out a some possible holes I could cut to let air flow. The new cooler weighs 258g, while the old one was 164g. It is also a bit heavier because the larger fan which also has a plastic frame.

It's not the prettiest, but I took two measurements for each heatsink nut from the motherboard stand-offs then located the new holes. I started with a step drilled hole, then sketched in the rest of the bracket and filed it out.

I had to trade the rubber feet out for slightly taller ones, but I have an idea for a slightly better solution. The current solution leaves exposed holes, and steel bits on the bottom that might scratch things. My plan is to get a piece of metal​/wood​/plastic, and mount it under the case with some little spacers, then attach the original rubber feet to that.

The fan situation is just a single 4-pin header that was split. The CPU fan was controlled by PWM, but the chassis fan got straight 12V.

I traded the 60mm chassis fan for a Gelid Silent 6, but since I couldn't control it, I used a 3-pin adapter with a resistor in-line (10 or 12 ohm I think) to cut the voltage a bit. After tuning the BIOS fan control this machine is no louder than the workstation it sits on.

Stock chassis fan - SunOn Maglev 0.89W!

Stock cooler and fans

New heatsink bracket stick out

Initial holes test fit

Holes filed to completely clear the cooler mount

New fans and better cable management

My workstation has a serial port header. This machine has 4 serial ports, and the ability to redirect the console to one or more. So I went to work putting a serial header in dreadnought which lead to a whole saga of my workstation breaking.

That put a nice damper on things so this sat for a few days.

The serial console is working great for accessing BIOS, but we need a proper OS install with serial support too. Why stop there though. We need to install the OS with the serial console.

What does that look like? You grab your regular install ISO, unpack it, modify some things, repack it, and you're off to the races. Right? No, you failed. We've got other things to do though so I did the install the old fashioned way.

If you want to read about my failure (or maybe my future success) you can check out: Installing with serial consoles

So the install is done, but we're not done discovering new ways to fail. Onwards to the next cool thing. Ansible configuration management. That's right, we're going to automate every last bit of configuring this machine from configuring serial ports, to docker, and frigate.

Our starting point is a machine with a configured network interface, and an SSH server, both of which were achieved by the installer.

Previously, I cut holes in the bottom of the case. They need to be covered to prevent scratching things, and to look nice.

I got some acrylic sheets. The plan was to use one with a cut-out to fit around the CPU cooler mounting bracket, then a second one to cover it up.

To cut them to size I used a steel straight edge, and a utility knife to score both sides. The first attempt failed with the break wandering. Next I scored more deeply, and clamped them to a table with a wood block, and used a second wood block to snap the piece off. The amount of force needed to snap them was still surprising. No pictures of the process.

For the middle cut out I just drilled a hole, and used the scroll saw to cut a window. The finish is rough, but that gets covered up.

The sheets are held on by the original feet with some longer M3 button head screws.

After scoring and snapping sheets

Hole cut-out using scroll saw

Spacer piece in place

Completed case bottom with rubber feet

Case front

Shiny case bottom

Not shiny edges

I wanted more memory, but didn't have it, or did I?

I had a single 16GB stick, and forbidden had a 16GB stick. I looked at them, and they are both Micron chips. The first stick was marked D9TBJ, and the other D9TZX.

ShareVDI/Vnopn single stick

Stick from forbidden (N100 laptop)

Micron has a FBGA parts decoder that will give you the part number. Then you can look up the data sheet, which lets you decode the last bit of the model number, and actually tell how fast your chips are.

Noting that these were both marked as DDR4-2666 you can see that cheap RAM sellers like to inflate numbers. Anyway, my old sticks were Transcend TS1GSH64V6B (DDR4-2666 CL19) so even the slower stick is a slight improvement.

When I loaded them up the machine chose DDR4-2666 CL19, so I ended up at the same speed, but now I've got 32GB.

Due to the limited space I needed some creative solutions for adding storage.

Initially I failed to find a low-profile NVMe + SATA combo PCIe adapter so I designed my own. I ordered some PCBs, but then I realized I forgot to adjust trace lengths. When I got the PCBs the prep by JLCPCB was a little disappointing. I made the PCB taller because it was a requirement for V-cut on the edge connector, but it looks like the cutter barely touched it.

It would have looked something like (missing M.2 connectors):

SATA side

PCIe side

Before I got around to ordering components I stumbled on Silverstone's SST-ECM28. It's a little pricy at $27, but I guess you can do that when you're the only one making such a thing. The components to complete my boards probably would have cost as much, and I might have needed to redo the board anyway. This seemed like a good compromise for now.

I stuck in a 1TB SATA SSD, and a 16GB Optane (mostly to verify functionality).

It looks like:

Optane on Silverstone PCIe adapter

Above view of installed PCIe M.2 adapter

I still need to find a suitable 2-4TB drive, which might end up being spinning rust.

This still leaves a free SATA port. If I want there to be a 2.5" drive I can cut a hole in the old drive mounting tray (for CPU fan clearance), and add it back.

This also leaves the Mini-PCIe, and M.2 E Key sockets to fill. They might get a Coral TPU, and 2.5GbE interface respectively.

I put the Coral TPU in. There's a software install and configuration guide at Coral.ai docs. Naturally things are out of date. The docs still reference the deprecated method for adding apt keys. The pycoral in the repository is incompatible with the newer python in Debian bookworm. There has been some movement through the past year, and even past week, on getting things updated: https://github.com/google-coral/pycoral/issues/137

Since I can't do the pycoral demo I guess I'll just test it when I get around to migrating frigate.

Mini PCIe Coral Edge TPU installed

With the last spare slot I added a M.2 RTL8125 based 2.5GbE adapter. This gives the machine 3 NICs. They are for regular net, camera network, and storage network.

This is just to enable fast bulk transfers when it comes to updating my video library.

I cut a little hole, and drilled a couple screw holes. Ended up having to clip the cover as well.

Card just fits next to the Mini-PCIe socket

Slighly too high jack mounting

I wall mounted pixy, and set up the ethernet connections in anticipation of adding some extra frigate cameras.

I was able to use the side foot mounts and a couple L brackets for a nice upside down wall mount. This keeps the fan intake at the bottom, and vent at the top for natural convection cooling. There is even enough space to sit the AC adapter on top without blocking the vents.

Mounting brackets with power supply

Ethernets + Dummy DisplayPort plug

Theoretically the migration should be kind of straightforward. I need to:

• migrate the docker containers and volumes
• export the zfs pool (external USB drive)
• shut down the old system
• import the zfs pool
• try to run the containers
• fix breakage
• migrate frigate storage to SSD
• add coral accelerator to frigate

Here is how it actually went:

Eventually I want this machine to output video on an HDMI port for feeding a Blonder Tongue HDE-CSV-QAM RF modulator.

I upgraded a bunch of machines to Debian 13 (trixie). This was probably the simplest one since it has fewer packages, and less configuration changes.

It broke the frigate container because the TPU driver wouldn't load.

The fix is to modify gasket-driver, but there's a caveat. The recommended modifications seem to be for kernel >= 6.13, however I've got 6.12.48.

I only needed to do the first sed:

sed -i -e 's/no_llseek/noop_llseek/g' gasket_core.c

The clue I got for this was from this commit by sethyx which was mentioned at Building gasket-dkms broken on linux kernel 6.5.0 (Ubuntu 23.10) #808.

This machine just dies sometimes. Nothing logged. Stays powered on.

According to the manual this has a NCT6116D hardware watchdog. It seems to be supported by the w83627hf_wdt watchdog timer module. I loaded the module, and in my dmesg output I got:

w83627hf_wdt: WDT driver for NCT6116 Super I/O chip initialising
w83627hf_wdt: initialized. timeout=60 sec (nowayout=0)

I added it to /etc/modules-load.d/modules.conf.

Then I installed the watchdog package, and modified /etc/watchdog.conf.

I uncommented the watchdog-device, and watchdog-timeout settings (left at default 60). Then rebooted, and set the BIOS timeout to 80 (just a little headroom).

To verify the watchdog was active, I sat on the GRUB screen for 80 seconds. It rebooted after a bit. Unfortunately it also rebooted once the kernel had loaded. I had enough time to check the dmesg output, and watchdog service. They showed:

NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
w83627hf_wdt: Watchdog already running. Resetting timeout to 60 sec

pixy watchdog[2023]: watchdog now set to 60 seconds
pixy watchdog[2023]: hardware watchdog identity: iTCO_wdt
pixy systemd[1]: Started watchdog.service - watchdog daemon.

I see that it is using the wrong watchdog (iTCO_wdt). I decided to blacklist that module since I'm not sure if ordering is guaranteed, and the service only seems to take the device name. I wasn't aware of this Intel TCO device until now. Not sure if there's a good reason to use one or the other, but I went ahead with my first choice.

Try again. Fail again. The module was still loaded. I needed to run depmod -ae, and update-initramfs -u to ensure the module does not get loaded.

Now watchdog service reports:

pixy watchdog[2100]: watchdog now set to 60 seconds
pixy watchdog[2100]: hardware watchdog identity: W83627HF Watchdog

The machine is alive and not rebooting. Now we trigger a kernel panic:

echo c > /proc/sysrq-trigger

The machine freezes, and after a while it reboots. Seems like everything is working.

Goal: Reduce heat output (and power usage). Spend as little as possible on the fewest new parts.

I came across some interesting DFI boards. These are TGU173-IB-1115GRE. A seller has a few listed for quite cheap. They are as-is, have no I/O plate, and only have a 4-pin power connector rather than the barrel jack.

They have 3 NICs, including one 2.5GbE. For expansion they have one 2280 M.2 PCIe, and one 2242 M.2 PCIe/SATA, plus a normal x4 PCIe slot. They use DDR4 SO-DIMMs, same as current.

I would have to get a M.2 2242 to Mini-PCIe adapter for the Coral TPU, but otherwise everything should tansfer over.

If money was no object I'd be considering an N305. Roughly the same single core, but more cores, and the same TDP. Lower power options like the N100 are nice, but they have a 16GB limit that isn't ideal.

Other interesting options are the 8-bay NAS boards with i5-8265U. They have a pair of 2.5GbE NICs, and a couple M.2 2280 slots. I would need one more gigabit NIC. Power is via 24-pin ATX connector so I would need a PicoPSU or other solution.

The day after I wrote this up, the seller dropped the price on two of the i3-1115GRE boards to $40, so I ordered one. Originally they had 1 of 3 Celeron 6305E versions at that price so I'm not sure what motivated the drop on the better CPU.

I ordered a blank I/O shield to cut up, and I'll have to see if I can cobble a barrel jack adapter together.

I got the new board. Managed to scrounge up a 4GB stick for testing. I stuck some loose pins on alligator clips and used my bench supply for now. First attempt it didn't POST, but I cleared CMOS and it came right up. I just ran Memtest86+ to verify basic functionality. It got through a couple passes.

This board seems to be a slightly custom order, or DFI made some updates. Along with the 4-pin connector (normal option), the CPU differs from the original spec. This has an i3-1115G4E. According to Intel, it lacks ECC support, and picks up an 'image processing unit' (compared to the GRE). Neither matter for my current purpose, but I would have liked ECC support.

Bench supply power setup

Memtest pass and CPU info

The only concerning parts of the board are some bent pins underneath the DisplayPort connector (no shorts), and some of the COM port headers and configuration jumpers got bent up.

I ordered another EDAC power supply that came with a new threaded jack socket. This one is the 60W model, EA10523C-120. Threads look different, but this at least leaves me with a supply for the original board.

I got the EDAC supply, and the plug and barrel jack are switchcraft. They are a water resistant variety with a o-ring seal, but it also has an unusually long barrel. My barrel jack power meter wouldn't work with the adapter because it barely reaches the internal switch.

I used a 4-pin ATX power extension for the motherboard connector. It uses the same key and pin out.

Assembled power adapter

Side view of mounted adapter

Above view of mounted adapter

The I/O shield has been a bit of an effort. I started by putting the blank shield in a case, and the motherboard in (no protruding connectors), then tracing around each port with a fine marker.

Tracing ports into the shield

Outlines from tracing

For the ethernets I picked a rough center point, drilled it big enough to fit my hand nibbler, and used a spare ethernet jack to quickly check my work. It turned out ok, but the double ethernet jack's center is slightly offset.

First drills and nibbles

Handy reference port

The HDMI didn't go well. I traced the full outline with a loose HDMI port. Next I drilled a starting hole, then tried scribing and flexing a section off. Finally I used my snips to wrench bits off. There just wasn't room for the nibbler at all. Eventually I looked at some other ideas, and came on a recommendation for a jeweler's saw. I've got one, but didn't think of it until now. I finished the HDMI with that, but it is still a bit rough.

HDMI attempt with 4 starting holes (fail)

Mangling the metal

The barrel jack was just a drilled hole. Step drill works great for those as long as you go slow on the last size, and go in from the back to clean up the edge.

USB C was just a pair of drilled holes, and two cuts with the jeweler's saw. Then filing to get a nice fit.

Half-way test fit

Plugging into the test fit

On the USB A ports I started to get a better feel for using the saw. It helps a lot to lay out the lines before hand. The saw needs to be held at an angle rather than perpendicular or else it just binds. The biggest frustration is frequently having to flip the blade to cut in a different direction, detaching a blade end to move to the next hole, then threading a new hole, re-tensioning, repeat. Final sizing is done with a fit check, a few sharpie marks, and steel jeweler's files.

It might be possible to do on the scroll saw, but I haven't really tried the fine blades there.

The USB A process

All ports test fit

All ports plug test

Plug test alternate view

Internal I/O shield view

The I/O shield was a bit wobbly to support the barrel jack. I went back after, and reinforced the barrel jack a bit. Just a bit of aluminum, and an extra screw, nut, and washer to fix it in place.

18.14.13

18.15.02

18.15.15

I still have plans to put in some serial ports, but I decided they aren't necessary at the moment. I don't want to take the special 2mm header ports off the other board. I'll wait until I can source a couple new ones.

I got hold of some 2.0mm ribbon cables with IDC connectors thinking I could assemble the ports myself. I tried fitting them on the board, but the header spacing is so tight that there is only room for the crimp style connectors side-by-side.

The other issue is I don't have any male solder serial ports suitable for panel mounting. I scoped out some potential bits, but it is still on hold.

I took down pixy, and debuilt it.

I went to fit the M.2 to Mini-PCIe adapter, and realized the matching height screw post is actually at 52mm, not 42mm. For 42mm it shares the short post below with the E key slot. I didn't have the right spacer and screw to bring it up. I ended up putting the adapter in the fancy Gen4 slot for now.

I was going to mount the Mini-PCIe part of the adapter on the drive cage, but my new CPU fan was too tall to install that without modification. My next plan was to drill and tap a hole in the bracket the cage mounts to, but it is too late for drilling. I taped it above the I/O area for now.

My idea to support the barrel jack with a screw and nut bit me. The nut blocks installation and removal of the 4-pin connector. Minor, but annoying.

The case fan wasn't oriented correctly for how I had the case wall mounted. Fortunately the cage is is reversible so that was an easy fix compared to fighting the soft fan mounts.

Power connector blocked

Machine running with tape job

I fixed up some of the things mentioned before:

• The CPU area now has a 32x32x6mm stack of thermal pad/putty under it. I also put some under the 2.5GbE chip.
• Pop rivet replaces the power jack screw. Power connector clearance is good
• I'm out of white silicone so I used kapton tape and some red filter tape to dim the Silverstone adapter's NVMe LED.
• The clock battery is fine. 3.01V
• The HDD activity LED is fine. Guess I just wasn't noticing it before.

Thermal pads (blue membrane was removed after)

Rivet installed and cut down

Power connector has good clearance now

Other changes:

• Moved the M.2 to Mini-PCIe adapter to the B key slot, and mounted the other part to the chassis.
• Fan speed changes
• TDP power limit changes

Spacer for M.2 mount

Mini-PCIe adapter mount 1

Mini-PCIe adapter mount 2

The fan speed control is basic. It just has 4 temperature boundaries, and 4 speed percentages. I set the boundaries to 40,45,55,60 (°C), and the speeds to 40,55,75,90 (%). Along with the thermal pads, and TDP changes I'm seeing base load temps of 40°C so that is keeping things much quieter. The fans stay around 1500 (51% fake PWM out of 120%) and 1800 RPM (58%).

In UEFI there are two options for TDP power limits. The first controls 'normal' operation, and the second controls turbo limitation for some period (Intel says it is configurable, but I guess DFI thought different). DFI had them set to 28W and 38W. I lowered them to 18W and 28W initially.

Unfortunately this hasn't made a big difference to wall power draw. I'm still seeing 22-24W with the base frigate load, and even up to 34W when Plex was busy.

Under load the CPU package seems to get just as hot, albeit more slowly, and can pull 38W from the wall. The CPU fan PWM tops out around 114% rather than 120%, but the tachometer reading is still 3600 RPM.

A bit dissatisfied with that, so I decided to try the 12W (TDP down). I couldn't find information about what the second power limit should be (DFI had it at 38W, and Techpowerup's 1115G4E page suggests it can be as high as 52W!). I decided to go with 18W.

Again, no difference to wall power draw, CPU temps or anything that I can see. Peak wall draw is 35W.

Stock power limits

My fan settings

My power limits

I played around with powercap-utils which seemed to report different values than I had set. Setting all the limits to 12W (except peak_power seems unable to go lower than 18W). Normal power draw seems unchanged, under full CPU load it seems to top out at 28W now.

Feels like I'm still missing something, or maybe my peripherals just need more power than I expected.

I made my cut-outs in the drive tray bracket (it has been out since I modded the case originally).

Rough fan clearance cutouts

After reading the motherboard manual more thoroughly it sounds like there is even less reason to have the bracket in place. It appears that enabling the second SATA port, disables both SATA and PCIe on the M.2 B key connector. To use the bracket now, I would have to ditch my M.2 SATA drive, or ditch the coral TPU.

I'd still like to have the bracket in place because it is one less thing to keep track of.

I forgot to come up with a design for the flat flex pass-through and Mini-PCIe mounting. Next day, I decided to just make another 35mm hole saw cut-out and line it with silicone edge protector. The Mini-PCIe bit just mounts to a new screw hole on the drive bracket, same general position.

Since that area stays warm, I put a couple tiny heatsinks on the gigabit ethernet chips. I'm not sure if the thermal pad on the 2.5G chip (under the board) is helping much.

Drive bracket installed

CPU fan holes close up

Drive cables secured for future use

Final back panel

I put it back on the wall mount. Hoping I can mostly ignore it now.

So far I haven't seen any PCIe errors with the Coral TPU like the previous system had. I was worried they would get worse because of the adapter and long flat flex cable.

The only issue seems to be sound related, but it hasn't caused any issues (not using the sound card at the moment), just some unusual dmesg output.

snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
snd_hda_intel 0000:00:1f.3: azx_get_response timeout, switching to polling mode: last cmd=0x20f70740
snd_hda_intel 0000:00:1f.3: No response from codec, disabling MSI: last cmd=0x20f70740
snd_hda_intel 0000:00:1f.3: azx_get_response timeout, switching to single_cmd mode: last cmd=0x20f70740
azx_single_send_cmd: 384 callbacks suppressed
azx_single_send_cmd: 267 callbacks suppressed

This might be power saving related. Could have started when I ran powertop to enable power saving features.