Going away drinks and burgers @ the OG ParlourOriginally published by DK on June 22, 2022 at 10:39 pm
Originally published by DK on June 21, 2022 at 9:28 am
Opening Day @ The HotelOriginally published by DK on June 1, 2022 at 8:44 pm
Waldmann Brewery and Wurstery, Hennepin County Library, Socca CafeOriginally published by DK on May 28, 2022 at 8:28 am
Thanks, Yagya ⚽️Originally published by DK on May 7, 2022 at 10:14 pm
Always nice to have treats at workOriginally published by DK on April 13, 2022 at 8:48 am
Not sure where this one came from (it was Wrecktangle)Originally published by DK on April 1, 2022 at 12:48 pm
Would probably be even better straight out of the ovenOriginally published by DK on March 19, 2022 at 12:21 am
Big thanks to our training manager for the Baker’s Wife donutsOriginally published by DK on March 1, 2022 at 8:37 am
Design is the spice of lifeOriginally published by DK on February 18, 2022 at 6:23 pm
The skyways are now openOriginally published by DK on February 12, 2022 at 4:45 pm
Thank you, Florian!Originally published by DK on February 7, 2022 at 6:53 pm
https://fourseasons.wd3.myworkdayjobs.com/en-US/search/job/Minneapolis/Assistant-IT-Manager_REQ10233293Originally published by DK on January 15, 2022 at 8:54 am
Work doge has two offices againOriginally published by DK on January 3, 2022 at 11:14 am
Thank you for all the kind hospitalityOriginally published by DK on December 10, 2021 at 8:57 pm
The spa sells logo merch!Originally published by DK on December 9, 2021 at 12:23 am
Such an eye-opening start to the tripOriginally published by DK on December 6, 2021 at 9:53 pm
Fun team visit to Cafe Alma (going back for birthday dinner on the fancy side next year)Originally published by DK on December 3, 2021 at 8:20 pm
Late work lunch with the crew @ Pizzeria Lola, MinneapolisOriginally published by DK on November 24, 2021 at 5:22 pm
https://www.fourseasons.com/minneapolis/Originally published by DK on November 19, 2021 at 12:21 am
The Market at Malcolm Yards, MinneapolisOriginally published by DK on November 12, 2021 at 10:21 pm
Way closer to the temp office than I thoughtOriginally published by DK on November 10, 2021 at 1:05 pm
Roti Modern Mediterranean, IDS Crystal Court, MinneapolisOriginally published by DK on November 9, 2021 at 1:26 pm
The small Apple + Pear @ Green + The Grain, MinneapolisOriginally published by DK on November 8, 2021 at 4:29 pm
A little late on writing the follow-up story to my post about the end of the stadium chapter, but as many of you already know, I’ve decided to return to the hospitality industry. On October 25th, I joined the opening team at the new Four Seasons Hotel and Private Residences Minneapolis, which will be located in the RBC Gateway building at the intersection of Nicollet Mall and Washington Avenue in downtown Minneapolis.
The project is owned by the Pohlad family, so a tip of the hat to Twins technology guru John Avenson for telling me about this opportunity (and the history behind it). The tower will be mixed use: office space for several Pohlad companies and RBC Wealth Management, a new ground floor restaurant and coffee shop from Gavin Kaysen, event space (two ballrooms and multiple meeting rooms), an outdoor pool deck with restaurant, spa with indoor pool, exercise and yoga rooms, eight hotel floors and 34 private residences.
The Four Seasons brand and corporate culture is outstanding and will be a great addition to the Twin Cities market. I’m so impressed with the local team being assembled here, along with all the people that I’ve met from the home office in Toronto and at other locations around the world. The Director of IT from Houston was in Minneapolis last week and will help with our opening. Later this month, I will travel to Chicago for orientation (and my first stay at one of the properties).
From a technical perspective, this move will take me back to the world of Opera, along with all of the other systems that hotels need to operate: door locks and keys, IPTV, WiFi, telephones, security cameras and digital signage (among others). It’s also really nice to have a strong network of internal support, as well as a world-class corporate IT group that have well-defined standards and procedures. My office will have a window peering into the data center and it’ll be fun to have IDF rooms on multiple floors of a really tall building (along with brand-new equipment throughout).
Similar to my start at the stadium, we are in temporary offices down the street while construction continues (offices should be ready in February, with the hotel set to open in the summer). Great to be back in construction mode again with a hard hat and reflective vest: McGough is the general contractor and I get to work with Gephart again too. Other primary groups include Pohlad-owned United Properties as the site developer and JLL as the leasing agent.Originally published by DK on November 7, 2021 at 11:59 pm
Team outing on UniversityOriginally published by DK on November 5, 2021 at 5:43 pm
First day at the new projectOriginally published by DK on October 25, 2021 at 11:08 pm
Yesterday was my last official day working at U.S. Bank Stadium as the Director of IT. I was about a week shy of six years, but what a wild ride it has been:
- Five plus seasons of Vikings Football (including the Minneapolis Miracle)
- Super Bowl LII
- Men’s Final Four
- Summer X Games
- A.C. Milan vs. Chelsea
- Monster Jam and Super Cross dirt shows
- MSHSL Championships
- Metallica, Coldplay, Taylor Swift, Guns N’ Roses, Beyoncé and Jay-Z, U2 with Beck (and many other shows)
It’s been an honor to work in this building from pre-opening through today, partnering with outstanding organizations, working with truly amazing people and learning an awful lot about the sports and entertainment industry.
There are really too many people to thank for helping me succeed in this chapter of my life, but I’d like to call out a few:
- Armon Farrari, who I stole from Apple (h/t Pash) and who became one of the best technology managers I’ve ever had the pleasure to work with in any industry.
- Curtis Schmillen, Director of Operations, Leo Pidde (now retired) and Ed Kroics (who took Leo’s place): just an amazing group of coworkers inside the building and great people to be with outside of work – be it on a golf course or at a happy hour.
- Jim Farstad, executive director of the Minnesota Sports Facility Authority (and former technology consultant during construction): a true renaissance man in every way.
- Inaugural GM Patrick Talty (and HR manager Shelly Wynia) for picking me to join the opening team back in 2015.
- Current GM John Drum, who helped during my health emergency outside the stadium in 2018.
- The CTAC team at CenturyLink (now Lumen) that designed and built the original technology infrastructure: Andy Diment, Mark Olson and Evan Teisberg. Still helping to this day, extraordinary knowledge that always made my job “really pretty simple.”
- Bill Anderson and Wesley Terry at AmpThink: no one in the world knows more about WiFi than these two individuals.
- Mentor Doug Podolak: your continued support of my career (and friendship) means the world to me.
- My wife and daughters: zero percent chance I’d be here today without their love and support. You da best ❤️
Lastly, to everyone at the Vikings (past and present): you’ve been great partners and I can’t wait to see you bring home the Vince Lombardi trophy to the great state of Minnesota. Skol, Vikings!Originally published by DK on October 21, 2021 at 10:50 pm
Emotional day at the stadium ❤️Originally published by DK on October 18, 2021 at 7:25 pm
Last Vikings game @ U.S. Bank Stadium: Vikings 19, Lions 17Originally published by DK on October 10, 2021 at 7:47 pm
https://recruiting.adp.com/srccar/public/RTI.home?c=1152751&d=ExternalCareerSite&r=5000780763706#/Originally published by DK on October 9, 2021 at 10:16 am
I was a skeptic, but it really is a cool effect ❄️Originally published by DK on October 1, 2021 at 3:40 pm
Should’ve put it on my calendar – missed all the chefsOriginally published by DK on September 23, 2021 at 6:30 pm
Thanks to Cisco for helping cross that one off the listOriginally published by DK on September 16, 2021 at 9:28 am
Work outing to Target Field with a stop @ Kieran’sOriginally published by DK on September 14, 2021 at 10:11 pm
And both at Target Field, no lessOriginally published by DK on August 24, 2021 at 9:29 am
Reverse ATMs and WiFi6 @ Target FieldOriginally published by DK on August 18, 2021 at 2:49 pm
The first of many, I’m sureOriginally published by DK on August 17, 2021 at 10:55 am
Release the hounds!Originally published by DK on August 14, 2021 at 10:04 am
MAKE IT LOUDOriginally published by DK on August 10, 2021 at 6:51 pm
Third time’s the charm, right?Originally published by DK on August 10, 2021 at 6:38 pm
Original Hockey Mom Brownies are coming to my office this fallOriginally published by DK on July 31, 2021 at 3:06 pm
…but the artwork is niceOriginally published by DK on July 6, 2021 at 4:27 pm
Why is my office always in the Star Tribune?Originally published by DK on July 5, 2021 at 8:24 am
Stadium Opening 2.0Originally published by DK on July 1, 2021 at 10:00 am
Rush Creek + WillingersOriginally published by DK on June 16, 2021 at 10:02 pm
Always good to get a refresher…Originally published by DK on June 14, 2021 at 2:39 pm
Events are coming back and U.S. Bank Stadium needs help getting ready. Looking for an entry-level tech support candidate to join my team this summer:
- Will train – great first job for a tech-savvy graduate
- NFL game day excitement at least ten days a year
- Good chance of working with roadies from world-famous bands
- Get paid to learn really expensive systems and equipment
- Flexible hybrid remote and onsite work environment
- Cool office that makes the news on a regular basis
- Fun team, decent boss
Days like these make it taste real goodOriginally published by DK on June 8, 2021 at 1:29 pm
Stadium deliveryOriginally published by DK on June 3, 2021 at 11:45 am
Ghosts of Final Fours 🏀Originally published by DK on May 25, 2021 at 4:15 pm
Getting ready for another goOriginally published by DK on May 18, 2021 at 3:48 pm
Lunch at Smokey’s – thanks, Jesse!Originally published by DK on April 28, 2021 at 2:20 pm
Not quite ready, but nearly thereOriginally published by DK on April 14, 2021 at 2:00 pm
Looking back through the archives, I don’t think I’ve ever actually written about our baseball ticket group. State Farm Insurance Agent Kirk Detlefsen (and his family) have been running a Twins season ticket group for quite some time. I was introduced to them through my Northwest Airlines mentor six years ago and have been hooked ever since.
Each season, Kirk buys up a bunch of season tickets in different sections, then divides them into shares at various price points. A ticket “draft” is held every year and each shareholder picks the games they want to attend. Usually this is held in February in the Delta Club with snow on the field, guest speakers from the Twins and an assortment of ballpark food available for purchase. We didn’t have one at all in 2020 and the 2021 version was held via Zoom (of course). Kudos to all the Detlefsens for pulling that off (and managing 100% digital tickets for the first time ever).
The smallest share in the group is two tickets to two games in the Champions Club. Since these seats are super expensive (and generally not available to the general public via single game sales), this was a nice way to splurge once a year (I split the share with my uncle). After a few years of that, we switched to a share with two seats for ten games in the Delta Club. That’s what we currently have, although it appears that our actual seats may be all over this season because of capacity restrictions.
So what was the return of fans like?
To be honest, the thing I was looking forward to most were the new Andrew Zimmern KFC wings that Stephanie March wrote about for Mpls/StPaul magazine. They are available in Bat & Barrel (which will now be called Truly On Deck, but none of the signage has changed yet), so we headed there first. They still apparently take advanced reservations, as most tables had reserved signs on them and you couldn’t order food from the counter (just drinks at the bar). I asked the person standing at the podium if I had to order through the app, but she wasn’t sure. The tables had QR codes on them, so I scanned that, which took me to a menu page, but no link to order online. Hmm…
So let’s talk about apps for a minute. MLB offers several, but these are the two primary ones I’ve used (and had on my phone): MLB and MLB Ballpark. I could’ve sworn the first one used to be called At Bat, but maybe I’m making that up. The MLB app lets you follow games around the league, while the Ballpark app is supposed to handle all the stuff you need at an actual game – tickets, mobile ordering, Twingo. I opened the wrong app so many times – maybe I need to move them to different screens. I also added the ParkWhiz app to my phone today to park in Ramp A, but more on that later.
There were multiple reports across the league of issues with mobile ordering on opening day, so I was interested to see how the process worked. I was also under the impression that all food needed to be ordered from the app, but that was not the case at all. With limited stands open, lines were long all around the main concourse, but you could place an order at a stand and pay with a credit card (no cash accepted – Apple Pay did work fine, which has been an issue for me there in the past). Here’s what the Ballpark app screens looked like to order my KFC wings:
The first disclaimer screen comes up every single time you try to order. The app doesn’t automatically try to figure out where you are located, so you have to select from the drop-down menu, which has what seems like a million options to scroll through. When I selected Bat & Barrel, there was a limited selection of items – I could order the wings, but no drinks or sides. The app was not integrated with Apple Pay, so I had to manually enter my credit card information. My card got billed right away and the screen said I would receive a text message when my order was ready. The app didn’t ask for my number, though, so I never received a text. Also, once you closed the confirmation window, it was not obvious how to pull up your order info (with the important order number) and I did not receive any email receipts of the transaction. Later I found that you can access a “My Orders” section, but that is only available by starting a new transaction:
There was signage at the front serving area for mobile order pickup, but there was no one working there. After a few minutes of standing around, someone came and asked me if I was waiting for an order, then went back to the kitchen to get it. By this time, there were a few other people waiting too. When they brought out my food, it was in a stapled shut brown grocery bag that just contained the food in a container – there were no utensils or napkins. This was only the second game back after a year of no fans, so hopefully these issues will get worked out as operations get back into the grove. The quality of the wings was really good, but a cheese brat I bought later from the Kramarczuk’s stand was small and overcooked.
While it was great to be back at a game, we had a few other negative experiences. The digital tickets in the app say you must enter at the gate indicated, so we headed to gate 34 when we arrived. Since we were early, there was no one in line at all, but a guest service staffer immediately confronted us. She said this was an ADA entrance and wanted to see our “Sweet Spot” card on the app before letting us proceed (?). That part of the app was not working and she eventually just let us proceed to the empty security screening line. Guessing this process will also get better as the season progresses.
The announced attendance for the game was 9,817 and the concourses never felt overcrowded. People in general followed the mask and distancing rules while moving around, but the concession lines were tight and the group of “bros” behind us decided that since they were drinking all game long, they didn’t need to wear their masks. At all. Seat spacing was similar to the Saints last season, alternating rows with four-seat blocks on both ends in one row with two-seats blocks in the middle of the next. Blocked seats were zip tied shut, but one group asked an usher if they were supposed to cut them to sit (they were in the wrong section). We didn’t see any ushers trying to enforce mask rules, the scoreboard and PA announcements were minimal and it would be nice if the fancam operators would only show people following the rules.
With state regulations capping attendance at 10,000 right now, I don’t envy the task of the ticket office managers. We really like our experience in Delta Club and hope that we can have seats back in there later in the season. It was rather frustrating to see entire empty sections up there the whole game, but I’m guessing that’s because most people wanted to stay inside the warm part of the club (which we would have traded our 12th row seats for in a heartbeat – it never got above 47F). Target Field’s current published capacity is only 38,544, so jumping up to 50% shouldn’t be a huge change. It will also be nice to see the menus expand back to normal – Red Cow, for example, was not offering turkey burgers yet.
The final issue we had (aside from the dumb start-a-runner-on-second rule and Twins loss) was leaving the parking ramp. I drove up to the exit kiosk and opened the ParkWhiz app, only to find my barcode gone and a prompt to login. When I bought the parking pass earlier in the day, they sent me an email with a link to the barcode, but never asked me to set up a login ID and password. With people waiting behind me (and no staff at the exit), I had to pull my car over to the side, find the email, then get back in line. I will now know to have this ready before I start driving, but I wasn’t the only car that had to do this. If there was a regular-sized crowd exiting at the same time, this would be a disaster.
A return to hockey is up next for me tomorrow night (with another Twins game on 4/23). Stay tuned for more reports…Originally published by DK on April 11, 2021 at 1:45 pm
First of four days @ USBSOriginally published by DK on April 8, 2021 at 9:28 am
10,000 more shots in four daysOriginally published by DK on April 7, 2021 at 10:02 pm
https://www.monitorsaintpaul.com/stories/tabitha-wheeler-named-social-worker-of-year,1741Originally published by DK on March 2, 2021 at 6:32 pm
The first of, I suspect, many changes comingOriginally published by DK on March 2, 2021 at 11:39 am
Working in technology usually means a fair amount of fixing stuff that’s broken. The advice of Roy and Moss from The IT Crowd is often handy: “Have you tried turning it off and on again?” Strong Google/DuckDuckGo skills and some free time are usually all you need to fix almost anything these days. I’ve really only had one situation in my tech career that had me stumped for a long time: bad networking at HSRA. That issue eventually got resolved with new hardware, but it’s a terrible feeling when technology doesn’t follow a logical troubleshooting process.
A week ago, another issue surfaced that may join the troubleshooting hall of shame. It started with a team member who couldn’t login to the file server on our office network over the weekend. With most of our staff working remotely, the on-premise file server is usually accessed via VPN, from both Mac and Windows clients. I connected from home and was able to see the file server (and thankfully all of the files) from my office iMac, but could not connect from my MacBook Pro. I find it useful to keep troubleshooting notes for future me, so please continue on if you’d like some nerd-tastic reading.
Like most everything in our building, things are starting to get old. The file server is part of the core infrastructure that was installed back in 2016. Our EMC VNXe3200 SAN is the foundation of a virtual environment with VMware hosts (Dell acquired EMC in 2016). There are a pair of Windows virtual servers, along with a Linux VM. The Windows servers act as our primary and secondary directory servers (Active Directory, still on Windows Server 2012), while the Linux box runs our intranet and some other IPTV services. It’s a fairly complex setup, but has been rock solid up to this point.
The VNXe3200 can serve CIFS shares directly, using AD for file permissions and access management. My AD servers are set to automatically install Windows updates, which I suspect was the root cause of this problem. The SAN hardware all seemed to be fine – no disk, power or network issues. People who were connected had no problems; it was looking like an issue with the authentication from AD. The web-based Unisphere management interface for the SAN was still running the Flash version of the Operating Environment (OE), so I needed to figure out a way around that issue to get more info from the logs (since Flash is now dead).
One of my original project engineers was able to set me up with a very old VM that had a copy of Firefox with the Flash plugin. Being careful to restrict network access for both security and auto-update reasons, I managed to get Unisphere updated to the latest OE version with HTML5 (220.127.116.1186894). Looking at the logs, the SAN had lost connectivity to directory services:
All Domain Controller servers configured for the CIFS server are not reachable. Please check this is not a network connectivity issue. Ensure at least one Domain Controller is up and running and is reachable by VNXe storage array.
It did not appear to be a network issue and I could login to the AD servers with no problem (with multiple user accounts). The support contract I had with Dell/EMC on the VNXe3200 expired and my experience trying to contact them did not go well. They took forever to respond and when they finally did, they wanted to charge me for all of the expired time, plus another year (for an amount that was about equal to just buying a new solution).
My immediate concern was making sure we had reliable backups of all files and folders. My ultra-low cost cloud backup strategy is to connect to the file server on my office iMac and use an app called qBackup that connects to a Backblaze B2 Cloud Storage account. This has worked great for years – the script runs nightly with an incremental backup, qBackup was a one-time cost of $30 and the monthly Backblaze charge is usually under $10 a month. The flaw in this cloud backup strategy is that it takes a really long time to restore 2.5 terabytes of data online (you can pay $189 and wait for Backblaze to ship a hard drive copy, but I don’t know how long that takes).
Since I still had access to the file server from the iMac, I stopped at Best Buy to get a 4TB external hard drive (and some thumb drives for people that needed files right away via SneakerNet). Now I had the cloud backup *and* locally attached copies of everything in my office. I tracked down the original engineer that helped install and configure this setup and he helped me create a new share that is served directly from the domain controller. I copied over all of the backup files from the 4TB drive and tested connections. That worked, so I reviewed security settings with our GM and applied permissions to folders via AD security groups. Cloud backup was re-pointed to the new share and ran successfully from my iMac. I created cheat sheets on how to connect to the new share from both Mac and Windows clients and sent them out to our staff. Done, right? Well…
All of my Mac users had no problem connecting and seeing what they were permitted to see. Some Windows users were also completely fine, but others connected and could not view all of the folders they were allowed to see. I initially had access based enumeration turned on, so some hidden folders were expected (but not ones they should see). One colleague saw all folders in the office, but not over VPN from home (on a brand-new laptop). This TechNote pointed towards a local cache issue (which this article also talks about). We’ve been experimenting with various offline settings and most people are now connected successfully. Here’s how we are troubleshooting Windows connections now:
- Restart the workstation first, Roy
- Re-map the network drive using a different letter
- Use the full AD name (i.e. – domain.local\username)
- Delete local offline cache files
Dell/EMC sent me a notice this weekend that there is another update to the Unisphere OE (18.104.22.16899487), but I didn’t see anything relevant in the release notes. I did download the huge .gpg file anyways, but so far the health check is timing out and I can’t get it installed. Also forgot to mention that we rolled back two of the automatic Windows Server updates that installed in mid-February (and turned off automatic updates). That obviously isn’t a long-term strategy, so I’d like to get updated to Windows Server 2019 soon (VMware updates too). OneDrive, Teams, SharePoint and even Box/Dropbox for Business are all options that may come into play as well.
Serving files shouldn’t be rocket science. At least I felt a little better when the engineer said, “I’ve installed hundreds, if not a thousand, file servers like this and I’ve never seen one do what yours is doing.”
Trailblazing!Originally published by DK on March 1, 2021 at 5:12 pm
Experienced some weird technical glitches this week:
- The older of my two AWS instances just stopped running right after midnight one night. Uptime Robot notified me right away via email, but I didn’t see it until around 8am, so there was some significant downtime (relatively speaking). The instance restarted fine, but I don’t know what caused that (and Amazon didn’t notify me of any issues).
- I made one late night attempt earlier this week to move this site to my newer AWS instance, but had two issues: 1) my nginx virtual host config is apparently wrong, as it started sending requests to a different domain after I updated DNS and 2) creating a new Let’s Encrypt certificate failed (probably because the domain name didn’t match). Decided to just revert back for now and make another run at it later.
- The iOS WordPress app update this week seems to have now broken all functionality with this site. Before, I just had issues uploading featured images (they would upload to the media library successfully, but the app would never get a “finished” acknowledgement). Now it doesn’t even get that far. I’ll just post from macOS machines for now, but that’s a pain for mobile posting away from the house.
- My office workstation wasn’t set to automatically restart after a power outage, so it’s been off for a week. I have other machines to use at home, but that machine does my automated network share backups to the cloud, so I needed to get that issue resolved in person today. I went to check status when I returned home and initially couldn’t see it, but that turned out to be an issue with the DHCP lease expiring. All good now.
- Not a glitch, but decided to take a look at my Twitter settings tonight. Dropped some dead accounts (actually really surprised at how many accounts I follow that haven’t been active for 100 days or more), unmuted some people and started looking for a new profile picture. Twitter remains my last social media account – can’t decide if it stays or goes this year. Leaning towards keeping it, with a renewed focus on pruning and more active curation of what I mute and what I follow. Might go private too.
Thankful for a long weekend – stay warm, Minnesota peeps…Originally published by DK on February 12, 2021 at 9:09 pm
-5F and (probably dropping)Originally published by DK on February 12, 2021 at 11:26 am
Exhale. L.A., you’re on the clockOriginally published by DK on February 7, 2021 at 10:27 pm
Flexibility (and safety) is great, but nothing beats in personOriginally published by DK on January 21, 2021 at 11:02 pm
…but a more organized messOriginally published by DK on January 21, 2021 at 12:35 pm
These things are so finickyOriginally published by DK on January 9, 2021 at 6:46 pm
My head is still spinning after returning home from the last home game of a very strange 2020 season:
- Games with fewer people aren’t as stressful overall, but it still takes me several hours to wind down after the final whistle blows.
- Immensely thankful everyone on our team stayed healthy all season.
- I don’t walk nearly as many steps on game day since we got a Cushman for IT.
- Five regular seasons already? 2016: 5-3, 2017: 7-1, 2018: 5-3, 2019: 6-2, 2020: 3-5
- The people in the Vikings organization are extraordinary and it’s a pleasure to work with them year after year.
- Three stadiums are now newer than us in the NFL, but next year will feel like opening the building all over again.
Just a few things to clean up this week in the building, then on to 2021…Originally published by DK on December 20, 2020 at 6:14 pm
Once I found the right screwdriver, it wasn’t too bad (new battery on a Dell XPS 15)Originally published by DK on December 16, 2020 at 12:35 pm
More perks from having a kid at CandylandOriginally published by DK on December 15, 2020 at 4:06 pm