7 Minute Miles

Burger Time

Posted Wednesday, June 22nd, 2022 10:39 pm GMT -5 in Food,Work at 10:39 PM

Going away drinks and burgers @ the OG Parlour

Originally published by DK on June 22, 2022 at 10:39 pm

Another great story from STR

Posted Tuesday, June 21st, 2022 09:28 am GMT -5 in Links,Technology,Work at 9:28 AM

Wi-Fi Scores 100X Data Increase at the Super Bowl over 10 Years for 20X Less Money than 5G

Originally published by DK on June 21, 2022 at 9:28 am

Day One ✅

Posted Wednesday, June 1st, 2022 08:44 pm GMT -5 in Nature,Work at 8:44 PM

Opening Day @ The Hotel

Originally published by DK on June 1, 2022 at 8:44 pm

The days are a blur

Posted Saturday, May 28th, 2022 08:28 am GMT -5 in Food,Work at 8:28 AM

Waldmann Brewery and Wurstery, Hennepin County Library, Socca Cafe

Originally published by DK on May 28, 2022 at 8:28 am

Soccer Night @ Allianz Field

Posted Saturday, May 7th, 2022 10:14 pm GMT -5 in Sports,Work at 10:14 PM

Thanks, Yagya ⚽️

Originally published by DK on May 7, 2022 at 10:14 pm

Thanks, Chef!

Posted Wednesday, April 13th, 2022 08:48 am GMT -5 in Food,Work at 8:48 AM

Always nice to have treats at work

Originally published by DK on April 13, 2022 at 8:48 am

Pretty solid 🍕

Posted Friday, April 1st, 2022 12:48 pm GMT -5 in Food,Work at 12:48 PM

Not sure where this one came from (it was Wrecktangle)

Originally published by DK on April 1, 2022 at 12:48 pm

Thumbs up on Wrecktangle Pizza

Posted Saturday, March 19th, 2022 12:21 am GMT -5 in Food,Work at 12:21 AM

Would probably be even better straight out of the oven

Originally published by DK on March 19, 2022 at 12:21 am

Nice morning surprise at work

Posted Tuesday, March 1st, 2022 08:37 am GMT -5 in Food,Work at 8:37 AM

Big thanks to our training manager for the Baker’s Wife donuts

Originally published by DK on March 1, 2022 at 8:37 am

Wall patterns

Posted Friday, February 18th, 2022 06:23 pm GMT -5 in Design,Work at 6:23 PM

Design is the spice of life

Originally published by DK on February 18, 2022 at 6:23 pm

YOU ARE HERE

Posted Saturday, February 12th, 2022 04:45 pm GMT -5 in Work at 4:45 PM

The skyways are now open

Originally published by DK on February 12, 2022 at 4:45 pm

This was a nice breakfast surprise

Posted Monday, February 7th, 2022 06:53 pm GMT -5 in Food,Work at 6:53 PM

Thank you, Florian!

Originally published by DK on February 7, 2022 at 6:53 pm

Assistant to the Assistant

Posted Saturday, January 15th, 2022 08:54 am GMT -5 in Links,Technology,Work at 8:54 AM

https://fourseasons.wd3.myworkdayjobs.com/en-US/search/job/Minneapolis/Assistant-IT-Manager_REQ10233293

Originally published by DK on January 15, 2022 at 8:54 am

Somebody is watching me work

Posted Monday, January 3rd, 2022 11:14 am GMT -5 in Family,Pets,Work at 11:14 AM

Work doge has two offices again

Originally published by DK on January 3, 2022 at 11:14 am

So long, Four Seasons LAX

Posted Friday, December 10th, 2021 08:57 pm GMT -5 in Travel,Work at 8:57 PM

Thank you for all the kind hospitality

Originally published by DK on December 10, 2021 at 8:57 pm

It’s the fashion

Posted Thursday, December 9th, 2021 12:23 am GMT -5 in Fashion,Shopping,Travel,Work at 12:23 AM

The spa sells logo merch!

Originally published by DK on December 9, 2021 at 12:23 am

Four Seasons Los Angeles

Posted Monday, December 6th, 2021 09:53 pm GMT -5 in Holidays,Travel,Work at 9:53 PM

Such an eye-opening start to the trip

Originally published by DK on December 6, 2021 at 9:53 pm

Alex Roberts, Part II

Posted Friday, December 3rd, 2021 08:20 pm GMT -5 in Food,Work at 8:20 PM

Fun team visit to Cafe Alma (going back for birthday dinner on the fancy side next year)

Originally published by DK on December 3, 2021 at 8:20 pm

M M M My Sha-Roni

Posted Wednesday, November 24th, 2021 05:22 pm GMT -5 in Food,Work at 5:22 PM

Late work lunch with the crew @ Pizzeria Lola, Minneapolis

Originally published by DK on November 24, 2021 at 5:22 pm

The Minneapolis pages are live

Posted Friday, November 19th, 2021 12:21 am GMT -5 in Business,Links,Work at 12:21 AM

https://www.fourseasons.com/minneapolis/

Originally published by DK on November 19, 2021 at 12:21 am

The Friday Adventures continue

Posted Friday, November 12th, 2021 10:21 pm GMT -5 in Food,Work at 10:21 PM

The Market at Malcolm Yards, Minneapolis

Originally published by DK on November 12, 2021 at 10:21 pm

Flippin’ good (fish) burger

Posted Wednesday, November 10th, 2021 01:05 pm GMT -5 in Food,Work at 1:05 PM

Way closer to the temp office than I thought

Originally published by DK on November 10, 2021 at 1:05 pm

Flip the Bird

Posted Tuesday, November 9th, 2021 01:26 pm GMT -5 in Food,Work at 1:26 PM

Roti Modern Mediterranean, IDS Crystal Court, Minneapolis

Originally published by DK on November 9, 2021 at 1:26 pm

Tried a new one for salad day

Posted Monday, November 8th, 2021 04:29 pm GMT -5 in Food,Work at 4:29 PM

The small Apple + Pear @ Green + The Grain, Minneapolis

Originally published by DK on November 8, 2021 at 4:29 pm

A return to hospitality

Posted Sunday, November 7th, 2021 11:59 pm GMT -5 in Business,Personal,Work at 11:59 PM

A little late on writing the follow-up story to my post about the end of the stadium chapter, but as many of you already know, I’ve decided to return to the hospitality industry. On October 25th, I joined the opening team at the new Four Seasons Hotel and Private Residences Minneapolis, which will be located in the RBC Gateway building at the intersection of Nicollet Mall and Washington Avenue in downtown Minneapolis.

The project is owned by the Pohlad family, so a tip of the hat to Twins technology guru John Avenson for telling me about this opportunity (and the history behind it). The tower will be mixed use: office space for several Pohlad companies and RBC Wealth Management, a new ground floor restaurant and coffee shop from Gavin Kaysen, event space (two ballrooms and multiple meeting rooms), an outdoor pool deck with restaurant, spa with indoor pool, exercise and yoga rooms, eight hotel floors and 34 private residences.

The Four Seasons brand and corporate culture is outstanding and will be a great addition to the Twin Cities market. I’m so impressed with the local team being assembled here, along with all the people that I’ve met from the home office in Toronto and at other locations around the world. The Director of IT from Houston was in Minneapolis last week and will help with our opening. Later this month, I will travel to Chicago for orientation (and my first stay at one of the properties).

From a technical perspective, this move will take me back to the world of Opera, along with all of the other systems that hotels need to operate: door locks and keys, IPTV, WiFi, telephones, security cameras and digital signage (among others). It’s also really nice to have a strong network of internal support, as well as a world-class corporate IT group that have well-defined standards and procedures. My office will have a window peering into the data center and it’ll be fun to have IDF rooms on multiple floors of a really tall building (along with brand-new equipment throughout).

Similar to my start at the stadium, we are in temporary offices down the street while construction continues (offices should be ready in February, with the hotel set to open in the summer). Great to be back in construction mode again with a hard hat and reflective vest: McGough is the general contractor and I get to work with Gephart again too. Other primary groups include Pohlad-owned United Properties as the site developer and JLL as the leasing agent.

Originally published by DK on November 7, 2021 at 11:59 pm

Legendary Spices Hot Pot

Posted Friday, November 5th, 2021 05:43 pm GMT -5 in Food,Work at 5:43 PM

Team outing on University

Originally published by DK on November 5, 2021 at 5:43 pm

By Appointment Only

Posted Monday, October 25th, 2021 11:08 pm GMT -5 in Business,Work at 11:08 PM

First day at the new project

Originally published by DK on October 25, 2021 at 11:08 pm

The stadium chapter concludes

Posted Thursday, October 21st, 2021 10:50 pm GMT -5 in Business,Family,Football,Friends,Personal,Work at 10:50 PM

Yesterday was my last official day working at U.S. Bank Stadium as the Director of IT. I was about a week shy of six years, but what a wild ride it has been:

  • Five plus seasons of Vikings Football (including the Minneapolis Miracle)
  • Super Bowl LII
  • Men’s Final Four
  • Summer X Games
  • A.C. Milan vs. Chelsea
  • Monster Jam and Super Cross dirt shows
  • MSHSL Championships
  • Metallica, Coldplay, Taylor Swift, Guns N’ Roses, Beyoncé and Jay-Z, U2 with Beck (and many other shows)

It’s been an honor to work in this building from pre-opening through today, partnering with outstanding organizations, working with truly amazing people and learning an awful lot about the sports and entertainment industry.

There are really too many people to thank for helping me succeed in this chapter of my life, but I’d like to call out a few:

  • Armon Farrari, who I stole from Apple (h/t Pash) and who became one of the best technology managers I’ve ever had the pleasure to work with in any industry.
  • Curtis Schmillen, Director of Operations, Leo Pidde (now retired) and Ed Kroics (who took Leo’s place): just an amazing group of coworkers inside the building and great people to be with outside of work – be it on a golf course or at a happy hour.
  • Jim Farstad, executive director of the Minnesota Sports Facility Authority (and former technology consultant during construction): a true renaissance man in every way.
  • Inaugural GM Patrick Talty (and HR manager Shelly Wynia) for picking me to join the opening team back in 2015.
  • Current GM John Drum, who helped during my health emergency outside the stadium in 2018.
  • The CTAC team at CenturyLink (now Lumen) that designed and built the original technology infrastructure: Andy Diment, Mark Olson and Evan Teisberg. Still helping to this day, extraordinary knowledge that always made my job “really pretty simple.”
  • Bill Anderson and Wesley Terry at AmpThink: no one in the world knows more about WiFi than these two individuals.
  • Mentor Doug Podolak: your continued support of my career (and friendship) means the world to me.
  • My wife and daughters: zero percent chance I’d be here today without their love and support. You da best ❤️

Lastly, to everyone at the Vikings (past and present): you’ve been great partners and I can’t wait to see you bring home the Vince Lombardi trophy to the great state of Minnesota. Skol, Vikings!

Originally published by DK on October 21, 2021 at 10:50 pm

One last go at press box popcorn

Posted Monday, October 18th, 2021 07:25 pm GMT -5 in Family,Food,Friends,Work at 7:25 PM

Emotional day at the stadium ❤️

Originally published by DK on October 18, 2021 at 7:25 pm

End of a purple era 💜🏈

Posted Sunday, October 10th, 2021 07:47 pm GMT -5 in Food,Football,Friends,Work at 7:47 PM

Last Vikings game @ U.S. Bank Stadium: Vikings 19, Lions 17

Originally published by DK on October 10, 2021 at 7:47 pm

Now Hiring – Inquire Within

Posted Saturday, October 9th, 2021 10:16 am GMT -5 in Business,Football,Links,Technology,Work at 10:16 AM

https://recruiting.adp.com/srccar/public/RTI.home?c=1152751&d=ExternalCareerSite&r=5000780763706#/

Originally published by DK on October 9, 2021 at 10:16 am

Testing the snow machines

Posted Friday, October 1st, 2021 03:40 pm GMT -5 in Football,Weather,Work at 3:40 PM

I was a skeptic, but it really is a cool effect ❄️

Originally published by DK on October 1, 2021 at 3:40 pm

Better late than never

Posted Thursday, September 23rd, 2021 06:30 pm GMT -5 in Food,Work at 6:30 PM

Should’ve put it on my calendar – missed all the chefs

Originally published by DK on September 23, 2021 at 6:30 pm

Sunny Shore Lunch with Keri

Posted Thursday, September 16th, 2021 01:52 pm GMT -5 in Food,Friends,Work at 1:52 PM

Farmers Kitchen + Bar

Originally published by DK on September 16, 2021 at 1:52 pm

Minneapolis Cider Co. ✅

Posted Thursday, September 16th, 2021 09:28 am GMT -5 in Food,Work at 9:28 AM

Thanks to Cisco for helping cross that one off the list

Originally published by DK on September 16, 2021 at 9:28 am

Baseball Night in Minneapolis

Posted Tuesday, September 14th, 2021 10:11 pm GMT -5 in Baseball,Food,Work at 10:11 PM

Work outing to Target Field with a stop @ Kieran’s

Originally published by DK on September 14, 2021 at 10:11 pm

See the other Billie Joe(l) ✅

Posted Tuesday, August 24th, 2021 09:29 am GMT -5 in Concerts,Music,Work at 9:29 AM

And both at Target Field, no less

Originally published by DK on August 24, 2021 at 9:29 am

Industry research

Posted Wednesday, August 18th, 2021 02:49 pm GMT -5 in Baseball,Business,Work at 2:49 PM

Reverse ATMs and WiFi6 @ Target Field

Originally published by DK on August 18, 2021 at 2:49 pm

Data center cleanup day #1

Posted Tuesday, August 17th, 2021 10:55 am GMT -5 in Technology,Work at 10:55 AM

The first of many, I’m sure

Originally published by DK on August 17, 2021 at 10:55 am

Here we go, 2021 Edition

Posted Saturday, August 14th, 2021 10:04 am GMT -5 in Football,Work at 10:04 AM

Release the hounds!

Originally published by DK on August 14, 2021 at 10:04 am

Winter is Coming

Posted Tuesday, August 10th, 2021 06:51 pm GMT -5 in Football,Work at 6:51 PM

MAKE IT LOUD

Originally published by DK on August 10, 2021 at 6:51 pm

Almost back to normal

Posted Tuesday, August 10th, 2021 06:38 pm GMT -5 in Business,Football,Work at 6:38 PM

Third time’s the charm, right?

Originally published by DK on August 10, 2021 at 6:38 pm

Cat’s out of the bag now

Posted Saturday, July 31st, 2021 03:06 pm GMT -5 in Business,Food,Football,Work at 3:06 PM

Original Hockey Mom Brownies are coming to my office this fall

Originally published by DK on July 31, 2021 at 3:06 pm

Pretty much live here now

Posted Tuesday, July 6th, 2021 04:27 pm GMT -5 in Art,Football,Work at 4:27 PM

…but the artwork is nice

Originally published by DK on July 6, 2021 at 4:27 pm

Curious Minnesota

Posted Monday, July 5th, 2021 08:24 am GMT -5 in Business,Football,News,Work at 8:24 AM

Why is my office always in the Star Tribune?

Originally published by DK on July 5, 2021 at 8:24 am

Getting the joint ready

Posted Thursday, July 1st, 2021 10:00 am GMT -5 in Football,Work at 10:00 AM

Stadium Opening 2.0

Originally published by DK on July 1, 2021 at 10:00 am

Busy day of networking

Posted Wednesday, June 16th, 2021 10:02 pm GMT -5 in Business,Food,Golf,Work at 10:02 PM

Rush Creek + Willingers

Originally published by DK on June 16, 2021 at 10:02 pm

Fire safety training day

Posted Monday, June 14th, 2021 02:39 pm GMT -5 in Work at 2:39 PM

Always good to get a refresher…

Originally published by DK on June 14, 2021 at 2:39 pm

Help Wanted at USBS

Posted Tuesday, June 8th, 2021 02:38 pm GMT -5 in Football,Technology,Work at 2:38 PM

Events are coming back and U.S. Bank Stadium needs help getting ready. Looking for an entry-level tech support candidate to join my team this summer:

  • Will train – great first job for a tech-savvy graduate
  • NFL game day excitement at least ten days a year
  • Good chance of working with roadies from world-famous bands
  • Get paid to learn really expensive systems and equipment
  • Flexible hybrid remote and onsite work environment
  • Cool office that makes the news on a regular basis
  • Fun team, decent boss

The posting is now up on the stadium website, while here is the direct link to the ADP hiring site (PDF here).

Originally published by DK on June 8, 2021 at 2:38 pm

Plain ice water 👍

Posted Tuesday, June 8th, 2021 01:29 pm GMT -5 in Food,Weather,Work at 1:29 PM

Days like these make it taste real good

Originally published by DK on June 8, 2021 at 1:29 pm

Time to restock the candy drawer

Posted Thursday, June 3rd, 2021 11:45 am GMT -5 in Food,Work at 11:45 AM

Stadium delivery

Originally published by DK on June 3, 2021 at 11:45 am

Look what’s back, back again

Posted Tuesday, May 25th, 2021 04:15 pm GMT -5 in Sports,Work at 4:15 PM

Ghosts of Final Fours 🏀

Originally published by DK on May 25, 2021 at 4:15 pm

Football looms 🏈

Posted Tuesday, May 18th, 2021 03:48 pm GMT -5 in Football,Technology,Work at 3:48 PM

Getting ready for another go

Originally published by DK on May 18, 2021 at 3:48 pm

Totally worth the drive

Posted Wednesday, April 28th, 2021 02:20 pm GMT -5 in Food,Friends,Work at 2:20 PM

Lunch at Smokey’s – thanks, Jesse!

Originally published by DK on April 28, 2021 at 2:20 pm

FIREPOWER 🔥

Posted Wednesday, April 14th, 2021 02:00 pm GMT -5 in Technology,Work at 2:00 PM

Not quite ready, but nearly there

Originally published by DK on April 14, 2021 at 2:00 pm

A Return to Baseball

Posted Sunday, April 11th, 2021 01:45 pm GMT -5 in Baseball,Food,Longform,Technology,Work at 1:45 PM

Looking back through the archives, I don’t think I’ve ever actually written about our baseball ticket group. State Farm Insurance Agent Kirk Detlefsen (and his family) have been running a Twins season ticket group for quite some time. I was introduced to them through my Northwest Airlines mentor six years ago and have been hooked ever since.

Each season, Kirk buys up a bunch of season tickets in different sections, then divides them into shares at various price points. A ticket “draft” is held every year and each shareholder picks the games they want to attend. Usually this is held in February in the Delta Club with snow on the field, guest speakers from the Twins and an assortment of ballpark food available for purchase. We didn’t have one at all in 2020 and the 2021 version was held via Zoom (of course). Kudos to all the Detlefsens for pulling that off (and managing 100% digital tickets for the first time ever).

The smallest share in the group is two tickets to two games in the Champions Club. Since these seats are super expensive (and generally not available to the general public via single game sales), this was a nice way to splurge once a year (I split the share with my uncle). After a few years of that, we switched to a share with two seats for ten games in the Delta Club. That’s what we currently have, although it appears that our actual seats may be all over this season because of capacity restrictions.

So what was the return of fans like?

To be honest, the thing I was looking forward to most were the new Andrew Zimmern KFC wings that Stephanie March wrote about for Mpls/StPaul magazine. They are available in Bat & Barrel (which will now be called Truly On Deck, but none of the signage has changed yet), so we headed there first. They still apparently take advanced reservations, as most tables had reserved signs on them and you couldn’t order food from the counter (just drinks at the bar). I asked the person standing at the podium if I had to order through the app, but she wasn’t sure. The tables had QR codes on them, so I scanned that, which took me to a menu page, but no link to order online. Hmm…

So let’s talk about apps for a minute. MLB offers several, but these are the two primary ones I’ve used (and had on my phone): MLB and MLB Ballpark. I could’ve sworn the first one used to be called At Bat, but maybe I’m making that up. The MLB app lets you follow games around the league, while the Ballpark app is supposed to handle all the stuff you need at an actual game – tickets, mobile ordering, Twingo. I opened the wrong app so many times – maybe I need to move them to different screens. I also added the ParkWhiz app to my phone today to park in Ramp A, but more on that later.

There were multiple reports across the league of issues with mobile ordering on opening day, so I was interested to see how the process worked. I was also under the impression that all food needed to be ordered from the app, but that was not the case at all. With limited stands open, lines were long all around the main concourse, but you could place an order at a stand and pay with a credit card (no cash accepted – Apple Pay did work fine, which has been an issue for me there in the past). Here’s what the Ballpark app screens looked like to order my KFC wings:

The first disclaimer screen comes up every single time you try to order. The app doesn’t automatically try to figure out where you are located, so you have to select from the drop-down menu, which has what seems like a million options to scroll through. When I selected Bat & Barrel, there was a limited selection of items – I could order the wings, but no drinks or sides. The app was not integrated with Apple Pay, so I had to manually enter my credit card information. My card got billed right away and the screen said I would receive a text message when my order was ready. The app didn’t ask for my number, though, so I never received a text. Also, once you closed the confirmation window, it was not obvious how to pull up your order info (with the important order number) and I did not receive any email receipts of the transaction. Later I found that you can access a “My Orders” section, but that is only available by starting a new transaction:

There was signage at the front serving area for mobile order pickup, but there was no one working there. After a few minutes of standing around, someone came and asked me if I was waiting for an order, then went back to the kitchen to get it. By this time, there were a few other people waiting too. When they brought out my food, it was in a stapled shut brown grocery bag that just contained the food in a container – there were no utensils or napkins. This was only the second game back after a year of no fans, so hopefully these issues will get worked out as operations get back into the grove. The quality of the wings was really good, but a cheese brat I bought later from the Kramarczuk’s stand was small and overcooked.

While it was great to be back at a game, we had a few other negative experiences. The digital tickets in the app say you must enter at the gate indicated, so we headed to gate 34 when we arrived. Since we were early, there was no one in line at all, but a guest service staffer immediately confronted us. She said this was an ADA entrance and wanted to see our “Sweet Spot” card on the app before letting us proceed (?). That part of the app was not working and she eventually just let us proceed to the empty security screening line. Guessing this process will also get better as the season progresses.

The announced attendance for the game was 9,817 and the concourses never felt overcrowded. People in general followed the mask and distancing rules while moving around, but the concession lines were tight and the group of “bros” behind us decided that since they were drinking all game long, they didn’t need to wear their masks. At all. Seat spacing was similar to the Saints last season, alternating rows with four-seat blocks on both ends in one row with two-seats blocks in the middle of the next. Blocked seats were zip tied shut, but one group asked an usher if they were supposed to cut them to sit (they were in the wrong section). We didn’t see any ushers trying to enforce mask rules, the scoreboard and PA announcements were minimal and it would be nice if the fancam operators would only show people following the rules.

With state regulations capping attendance at 10,000 right now, I don’t envy the task of the ticket office managers. We really like our experience in Delta Club and hope that we can have seats back in there later in the season. It was rather frustrating to see entire empty sections up there the whole game, but I’m guessing that’s because most people wanted to stay inside the warm part of the club (which we would have traded our 12th row seats for in a heartbeat – it never got above 47F). Target Field’s current published capacity is only 38,544, so jumping up to 50% shouldn’t be a huge change. It will also be nice to see the menus expand back to normal – Red Cow, for example, was not offering turkey burgers yet.

The final issue we had (aside from the dumb start-a-runner-on-second rule and Twins loss) was leaving the parking ramp. I drove up to the exit kiosk and opened the ParkWhiz app, only to find my barcode gone and a prompt to login. When I bought the parking pass earlier in the day, they sent me an email with a link to the barcode, but never asked me to set up a login ID and password. With people waiting behind me (and no staff at the exit), I had to pull my car over to the side, find the email, then get back in line. I will now know to have this ready before I start driving, but I wasn’t the only car that had to do this. If there was a regular-sized crowd exiting at the same time, this would be a disaster.

A return to hockey is up next for me tomorrow night (with another Twins game on 4/23). Stay tuned for more reports…

Originally published by DK on April 11, 2021 at 1:45 pm

Ready for the jabs

Posted Thursday, April 8th, 2021 09:28 am GMT -5 in Football,Science,Work at 9:28 AM

First of four days @ USBS

Originally published by DK on April 8, 2021 at 9:28 am

Excited to help this weekend

Posted Wednesday, April 7th, 2021 10:02 pm GMT -5 in Football,Science,Work at 10:02 PM

10,000 more shots in four days

Originally published by DK on April 7, 2021 at 10:02 pm

Well-deserved recognition

Posted Tuesday, March 2nd, 2021 06:32 pm GMT -5 in Friends,Schools,Work at 6:32 PM

https://www.monitorsaintpaul.com/stories/tabitha-wheeler-named-social-worker-of-year,1741

Originally published by DK on March 2, 2021 at 6:32 pm

Katie, NO MOR 🥲

Posted Tuesday, March 2nd, 2021 11:39 am GMT -5 in Friends,Work at 11:39 AM

The first of, I suspect, many changes coming

Originally published by DK on March 2, 2021 at 11:39 am

File Server Blues

Posted Monday, March 1st, 2021 05:12 pm GMT -5 in Longform,Technology,Work at 5:12 PM

Working in technology usually means a fair amount of fixing stuff that’s broken. The advice of Roy and Moss from The IT Crowd is often handy: “Have you tried turning it off and on again?” Strong Google/DuckDuckGo skills and some free time are usually all you need to fix almost anything these days. I’ve really only had one situation in my tech career that had me stumped for a long time: bad networking at HSRA. That issue eventually got resolved with new hardware, but it’s a terrible feeling when technology doesn’t follow a logical troubleshooting process.

A week ago, another issue surfaced that may join the troubleshooting hall of shame. It started with a team member who couldn’t login to the file server on our office network over the weekend. With most of our staff working remotely, the on-premise file server is usually accessed via VPN, from both Mac and Windows clients. I connected from home and was able to see the file server (and thankfully all of the files) from my office iMac, but could not connect from my MacBook Pro. I find it useful to keep troubleshooting notes for future me, so please continue on if you’d like some nerd-tastic reading.

Like most everything in our building, things are starting to get old. The file server is part of the core infrastructure that was installed back in 2016. Our EMC VNXe3200 SAN is the foundation of a virtual environment with VMware hosts (Dell acquired EMC in 2016). There are a pair of Windows virtual servers, along with a Linux VM. The Windows servers act as our primary and secondary directory servers (Active Directory, still on Windows Server 2012), while the Linux box runs our intranet and some other IPTV services. It’s a fairly complex setup, but has been rock solid up to this point.

The VNXe3200 can serve CIFS shares directly, using AD for file permissions and access management. My AD servers are set to automatically install Windows updates, which I suspect was the root cause of this problem. The SAN hardware all seemed to be fine – no disk, power or network issues. People who were connected had no problems; it was looking like an issue with the authentication from AD. The web-based Unisphere management interface for the SAN was still running the Flash version of the Operating Environment (OE), so I needed to figure out a way around that issue to get more info from the logs (since Flash is now dead).

One of my original project engineers was able to set me up with a very old VM that had a copy of Firefox with the Flash plugin. Being careful to restrict network access for both security and auto-update reasons, I managed to get Unisphere updated to the latest OE version with HTML5 (3.1.12.10186894). Looking at the logs, the SAN had lost connectivity to directory services:

All Domain Controller servers configured for the CIFS server are not reachable. Please check this is not a network connectivity issue. Ensure at least one Domain Controller is up and running and is reachable by VNXe storage array.

It did not appear to be a network issue and I could login to the AD servers with no problem (with multiple user accounts). The support contract I had with Dell/EMC on the VNXe3200 expired and my experience trying to contact them did not go well. They took forever to respond and when they finally did, they wanted to charge me for all of the expired time, plus another year (for an amount that was about equal to just buying a new solution).

My immediate concern was making sure we had reliable backups of all files and folders. My ultra-low cost cloud backup strategy is to connect to the file server on my office iMac and use an app called qBackup that connects to a Backblaze B2 Cloud Storage account. This has worked great for years – the script runs nightly with an incremental backup, qBackup was a one-time cost of $30 and the monthly Backblaze charge is usually under $10 a month. The flaw in this cloud backup strategy is that it takes a really long time to restore 2.5 terabytes of data online (you can pay $189 and wait for Backblaze to ship a hard drive copy, but I don’t know how long that takes).

Since I still had access to the file server from the iMac, I stopped at Best Buy to get a 4TB external hard drive (and some thumb drives for people that needed files right away via SneakerNet). Now I had the cloud backup *and* locally attached copies of everything in my office. I tracked down the original engineer that helped install and configure this setup and he helped me create a new share that is served directly from the domain controller. I copied over all of the backup files from the 4TB drive and tested connections. That worked, so I reviewed security settings with our GM and applied permissions to folders via AD security groups. Cloud backup was re-pointed to the new share and ran successfully from my iMac. I created cheat sheets on how to connect to the new share from both Mac and Windows clients and sent them out to our staff. Done, right? Well…

All of my Mac users had no problem connecting and seeing what they were permitted to see. Some Windows users were also completely fine, but others connected and could not view all of the folders they were allowed to see. I initially had access based enumeration turned on, so some hidden folders were expected (but not ones they should see). One colleague saw all folders in the office, but not over VPN from home (on a brand-new laptop). This TechNote pointed towards a local cache issue (which this article also talks about). We’ve been experimenting with various offline settings and most people are now connected successfully. Here’s how we are troubleshooting Windows connections now:

  • Restart the workstation first, Roy
  • Re-map the network drive using a different letter
  • Use the full AD name (i.e. – domain.local\username)
  • Delete local offline cache files

Dell/EMC sent me a notice this weekend that there is another update to the Unisphere OE (3.1.14.10199487), but I didn’t see anything relevant in the release notes. I did download the huge .gpg file anyways, but so far the health check is timing out and I can’t get it installed. Also forgot to mention that we rolled back two of the automatic Windows Server updates that installed in mid-February (and turned off automatic updates). That obviously isn’t a long-term strategy, so I’d like to get updated to Windows Server 2019 soon (VMware updates too). OneDrive, Teams, SharePoint and even Box/Dropbox for Business are all options that may come into play as well.

Serving files shouldn’t be rocket science. At least I felt a little better when the engineer said, “I’ve installed hundreds, if not a thousand, file servers like this and I’ve never seen one do what yours is doing.”

Trailblazing!

Originally published by DK on March 1, 2021 at 5:12 pm

A Week of Gremlins

Posted Friday, February 12th, 2021 09:09 pm GMT -5 in Longform,Technology,Work at 9:09 PM

Experienced some weird technical glitches this week:

  • The older of my two AWS instances just stopped running right after midnight one night. Uptime Robot notified me right away via email, but I didn’t see it until around 8am, so there was some significant downtime (relatively speaking). The instance restarted fine, but I don’t know what caused that (and Amazon didn’t notify me of any issues).
  • I made one late night attempt earlier this week to move this site to my newer AWS instance, but had two issues: 1) my nginx virtual host config is apparently wrong, as it started sending requests to a different domain after I updated DNS and 2) creating a new Let’s Encrypt certificate failed (probably because the domain name didn’t match). Decided to just revert back for now and make another run at it later.
  • The iOS WordPress app update this week seems to have now broken all functionality with this site. Before, I just had issues uploading featured images (they would upload to the media library successfully, but the app would never get a “finished” acknowledgement). Now it doesn’t even get that far. I’ll just post from macOS machines for now, but that’s a pain for mobile posting away from the house.
  • My office workstation wasn’t set to automatically restart after a power outage, so it’s been off for a week. I have other machines to use at home, but that machine does my automated network share backups to the cloud, so I needed to get that issue resolved in person today. I went to check status when I returned home and initially couldn’t see it, but that turned out to be an issue with the DHCP lease expiring. All good now.
  • Not a glitch, but decided to take a look at my Twitter settings tonight. Dropped some dead accounts (actually really surprised at how many accounts I follow that haven’t been active for 100 days or more), unmuted some people and started looking for a new profile picture. Twitter remains my last social media account – can’t decide if it stays or goes this year. Leaning towards keeping it, with a renewed focus on pruning and more active curation of what I mute and what I follow. Might go private too.

Thankful for a long weekend – stay warm, Minnesota peeps…

Originally published by DK on February 12, 2021 at 9:09 pm

Looks nice out (but really not)

Posted Friday, February 12th, 2021 11:26 am GMT -5 in Weather,Work at 11:26 AM

-5F and (probably dropping)

Originally published by DK on February 12, 2021 at 11:26 am

Cheers to the staff in Tampa 🏈

Posted Sunday, February 7th, 2021 10:27 pm GMT -5 in Football,History,Work at 10:27 PM

Exhale. L.A., you’re on the clock

Originally published by DK on February 7, 2021 at 10:27 pm

Nothing beats the actual office

Posted Thursday, January 21st, 2021 11:02 pm GMT -5 in Baseball,Football,Technology,Work at 11:02 PM

Flexibility (and safety) is great, but nothing beats in person

Originally published by DK on January 21, 2021 at 11:02 pm

Still an absolute mess

Posted Thursday, January 21st, 2021 12:35 pm GMT -5 in Housekeeping,Technology,Work at 12:35 PM

…but a more organized mess

Originally published by DK on January 21, 2021 at 12:35 pm

Dell battery management 😡

Posted Saturday, January 9th, 2021 06:46 pm GMT -5 in Technology,Work at 6:46 PM

These things are so finicky

Originally published by DK on January 9, 2021 at 6:46 pm

Reflections on Season Five

Posted Sunday, December 20th, 2020 06:14 pm GMT -5 in Football,Mini Posts,Technology,Work at 6:14 PM

My head is still spinning after returning home from the last home game of a very strange 2020 season:

  • Games with fewer people aren’t as stressful overall, but it still takes me several hours to wind down after the final whistle blows.
  • Immensely thankful everyone on our team stayed healthy all season.
  • I don’t walk nearly as many steps on game day since we got a Cushman for IT.
  • Five regular seasons already? 2016: 5-3, 2017: 7-1, 2018: 5-3, 2019: 6-2, 2020: 3-5
  • The people in the Vikings organization are extraordinary and it’s a pleasure to work with them year after year.
  • Three stadiums are now newer than us in the NFL, but next year will feel like opening the building all over again.

Just a few things to clean up this week in the building, then on to 2021…

Originally published by DK on December 20, 2020 at 6:14 pm

Today’s brain surgery

Posted Wednesday, December 16th, 2020 12:35 pm GMT -5 in Technology,Work at 12:35 PM

Once I found the right screwdriver, it wasn’t too bad (new battery on a Dell XPS 15)

Originally published by DK on December 16, 2020 at 12:35 pm

White cheddar is a keeper

Posted Tuesday, December 15th, 2020 04:06 pm GMT -5 in Family,Food,Work at 4:06 PM

More perks from having a kid at Candyland

Originally published by DK on December 15, 2020 at 4:06 pm