Original Hockey Mom Brownies are coming to my office this fallOriginally published by DK on July 31, 2021 at 3:06 pm
…but the artwork is niceOriginally published by DK on July 6, 2021 at 4:27 pm
Why is my office always in the Star Tribune?Originally published by DK on July 5, 2021 at 8:24 am
Stadium Opening 2.0Originally published by DK on July 1, 2021 at 10:00 am
Rush Creek + WillingersOriginally published by DK on June 16, 2021 at 10:02 pm
Always good to get a refresher…Originally published by DK on June 14, 2021 at 2:39 pm
Events are coming back and U.S. Bank Stadium needs help getting ready. Looking for an entry-level tech support candidate to join my team this summer:
- Will train – great first job for a tech-savvy graduate
- NFL game day excitement at least ten days a year
- Good chance of working with roadies from world-famous bands
- Get paid to learn really expensive systems and equipment
- Flexible hybrid remote and onsite work environment
- Cool office that makes the news on a regular basis
- Fun team, decent boss
Days like these make it taste real goodOriginally published by DK on June 8, 2021 at 1:29 pm
Stadium deliveryOriginally published by DK on June 3, 2021 at 11:45 am
Ghosts of Final Fours 🏀Originally published by DK on May 25, 2021 at 4:15 pm
Getting ready for another goOriginally published by DK on May 18, 2021 at 3:48 pm
Lunch at Smokey’s – thanks, Jesse!Originally published by DK on April 28, 2021 at 2:20 pm
Not quite ready, but nearly thereOriginally published by DK on April 14, 2021 at 2:00 pm
Looking back through the archives, I don’t think I’ve ever actually written about our baseball ticket group. State Farm Insurance Agent Kirk Detlefsen (and his family) have been running a Twins season ticket group for quite some time. I was introduced to them through my Northwest Airlines mentor six years ago and have been hooked ever since.
Each season, Kirk buys up a bunch of season tickets in different sections, then divides them into shares at various price points. A ticket “draft” is held every year and each shareholder picks the games they want to attend. Usually this is held in February in the Delta Club with snow on the field, guest speakers from the Twins and an assortment of ballpark food available for purchase. We didn’t have one at all in 2020 and the 2021 version was held via Zoom (of course). Kudos to all the Detlefsens for pulling that off (and managing 100% digital tickets for the first time ever).
The smallest share in the group is two tickets to two games in the Champions Club. Since these seats are super expensive (and generally not available to the general public via single game sales), this was a nice way to splurge once a year (I split the share with my uncle). After a few years of that, we switched to a share with two seats for ten games in the Delta Club. That’s what we currently have, although it appears that our actual seats may be all over this season because of capacity restrictions.
So what was the return of fans like?
To be honest, the thing I was looking forward to most were the new Andrew Zimmern KFC wings that Stephanie March wrote about for Mpls/StPaul magazine. They are available in Bat & Barrel (which will now be called Truly On Deck, but none of the signage has changed yet), so we headed there first. They still apparently take advanced reservations, as most tables had reserved signs on them and you couldn’t order food from the counter (just drinks at the bar). I asked the person standing at the podium if I had to order through the app, but she wasn’t sure. The tables had QR codes on them, so I scanned that, which took me to a menu page, but no link to order online. Hmm…
So let’s talk about apps for a minute. MLB offers several, but these are the two primary ones I’ve used (and had on my phone): MLB and MLB Ballpark. I could’ve sworn the first one used to be called At Bat, but maybe I’m making that up. The MLB app lets you follow games around the league, while the Ballpark app is supposed to handle all the stuff you need at an actual game – tickets, mobile ordering, Twingo. I opened the wrong app so many times – maybe I need to move them to different screens. I also added the ParkWhiz app to my phone today to park in Ramp A, but more on that later.
There were multiple reports across the league of issues with mobile ordering on opening day, so I was interested to see how the process worked. I was also under the impression that all food needed to be ordered from the app, but that was not the case at all. With limited stands open, lines were long all around the main concourse, but you could place an order at a stand and pay with a credit card (no cash accepted – Apple Pay did work fine, which has been an issue for me there in the past). Here’s what the Ballpark app screens looked like to order my KFC wings:
The first disclaimer screen comes up every single time you try to order. The app doesn’t automatically try to figure out where you are located, so you have to select from the drop-down menu, which has what seems like a million options to scroll through. When I selected Bat & Barrel, there was a limited selection of items – I could order the wings, but no drinks or sides. The app was not integrated with Apple Pay, so I had to manually enter my credit card information. My card got billed right away and the screen said I would receive a text message when my order was ready. The app didn’t ask for my number, though, so I never received a text. Also, once you closed the confirmation window, it was not obvious how to pull up your order info (with the important order number) and I did not receive any email receipts of the transaction. Later I found that you can access a “My Orders” section, but that is only available by starting a new transaction:
There was signage at the front serving area for mobile order pickup, but there was no one working there. After a few minutes of standing around, someone came and asked me if I was waiting for an order, then went back to the kitchen to get it. By this time, there were a few other people waiting too. When they brought out my food, it was in a stapled shut brown grocery bag that just contained the food in a container – there were no utensils or napkins. This was only the second game back after a year of no fans, so hopefully these issues will get worked out as operations get back into the grove. The quality of the wings was really good, but a cheese brat I bought later from the Kramarczuk’s stand was small and overcooked.
While it was great to be back at a game, we had a few other negative experiences. The digital tickets in the app say you must enter at the gate indicated, so we headed to gate 34 when we arrived. Since we were early, there was no one in line at all, but a guest service staffer immediately confronted us. She said this was an ADA entrance and wanted to see our “Sweet Spot” card on the app before letting us proceed (?). That part of the app was not working and she eventually just let us proceed to the empty security screening line. Guessing this process will also get better as the season progresses.
The announced attendance for the game was 9,817 and the concourses never felt overcrowded. People in general followed the mask and distancing rules while moving around, but the concession lines were tight and the group of “bros” behind us decided that since they were drinking all game long, they didn’t need to wear their masks. At all. Seat spacing was similar to the Saints last season, alternating rows with four-seat blocks on both ends in one row with two-seats blocks in the middle of the next. Blocked seats were zip tied shut, but one group asked an usher if they were supposed to cut them to sit (they were in the wrong section). We didn’t see any ushers trying to enforce mask rules, the scoreboard and PA announcements were minimal and it would be nice if the fancam operators would only show people following the rules.
With state regulations capping attendance at 10,000 right now, I don’t envy the task of the ticket office managers. We really like our experience in Delta Club and hope that we can have seats back in there later in the season. It was rather frustrating to see entire empty sections up there the whole game, but I’m guessing that’s because most people wanted to stay inside the warm part of the club (which we would have traded our 12th row seats for in a heartbeat – it never got above 47F). Target Field’s current published capacity is only 38,544, so jumping up to 50% shouldn’t be a huge change. It will also be nice to see the menus expand back to normal – Red Cow, for example, was not offering turkey burgers yet.
The final issue we had (aside from the dumb start-a-runner-on-second rule and Twins loss) was leaving the parking ramp. I drove up to the exit kiosk and opened the ParkWhiz app, only to find my barcode gone and a prompt to login. When I bought the parking pass earlier in the day, they sent me an email with a link to the barcode, but never asked me to set up a login ID and password. With people waiting behind me (and no staff at the exit), I had to pull my car over to the side, find the email, then get back in line. I will now know to have this ready before I start driving, but I wasn’t the only car that had to do this. If there was a regular-sized crowd exiting at the same time, this would be a disaster.
A return to hockey is up next for me tomorrow night (with another Twins game on 4/23). Stay tuned for more reports…Originally published by DK on April 11, 2021 at 1:45 pm
First of four days @ USBSOriginally published by DK on April 8, 2021 at 9:28 am
10,000 more shots in four daysOriginally published by DK on April 7, 2021 at 10:02 pm
https://www.monitorsaintpaul.com/stories/tabitha-wheeler-named-social-worker-of-year,1741Originally published by DK on March 2, 2021 at 6:32 pm
The first of, I suspect, many changes comingOriginally published by DK on March 2, 2021 at 11:39 am
Working in technology usually means a fair amount of fixing stuff that’s broken. The advice of Roy and Moss from The IT Crowd is often handy: “Have you tried turning it off and on again?” Strong Google/DuckDuckGo skills and some free time are usually all you need to fix almost anything these days. I’ve really only had one situation in my tech career that had me stumped for a long time: bad networking at HSRA. That issue eventually got resolved with new hardware, but it’s a terrible feeling when technology doesn’t follow a logical troubleshooting process.
A week ago, another issue surfaced that may join the troubleshooting hall of shame. It started with a team member who couldn’t login to the file server on our office network over the weekend. With most of our staff working remotely, the on-premise file server is usually accessed via VPN, from both Mac and Windows clients. I connected from home and was able to see the file server (and thankfully all of the files) from my office iMac, but could not connect from my MacBook Pro. I find it useful to keep troubleshooting notes for future me, so please continue on if you’d like some nerd-tastic reading.
Like most everything in our building, things are starting to get old. The file server is part of the core infrastructure that was installed back in 2016. Our EMC VNXe3200 SAN is the foundation of a virtual environment with VMware hosts (Dell acquired EMC in 2016). There are a pair of Windows virtual servers, along with a Linux VM. The Windows servers act as our primary and secondary directory servers (Active Directory, still on Windows Server 2012), while the Linux box runs our intranet and some other IPTV services. It’s a fairly complex setup, but has been rock solid up to this point.
The VNXe3200 can serve CIFS shares directly, using AD for file permissions and access management. My AD servers are set to automatically install Windows updates, which I suspect was the root cause of this problem. The SAN hardware all seemed to be fine – no disk, power or network issues. People who were connected had no problems; it was looking like an issue with the authentication from AD. The web-based Unisphere management interface for the SAN was still running the Flash version of the Operating Environment (OE), so I needed to figure out a way around that issue to get more info from the logs (since Flash is now dead).
One of my original project engineers was able to set me up with a very old VM that had a copy of Firefox with the Flash plugin. Being careful to restrict network access for both security and auto-update reasons, I managed to get Unisphere updated to the latest OE version with HTML5 (188.8.131.5286894). Looking at the logs, the SAN had lost connectivity to directory services:
All Domain Controller servers configured for the CIFS server are not reachable. Please check this is not a network connectivity issue. Ensure at least one Domain Controller is up and running and is reachable by VNXe storage array.
It did not appear to be a network issue and I could login to the AD servers with no problem (with multiple user accounts). The support contract I had with Dell/EMC on the VNXe3200 expired and my experience trying to contact them did not go well. They took forever to respond and when they finally did, they wanted to charge me for all of the expired time, plus another year (for an amount that was about equal to just buying a new solution).
My immediate concern was making sure we had reliable backups of all files and folders. My ultra-low cost cloud backup strategy is to connect to the file server on my office iMac and use an app called qBackup that connects to a Backblaze B2 Cloud Storage account. This has worked great for years – the script runs nightly with an incremental backup, qBackup was a one-time cost of $30 and the monthly Backblaze charge is usually under $10 a month. The flaw in this cloud backup strategy is that it takes a really long time to restore 2.5 terabytes of data online (you can pay $189 and wait for Backblaze to ship a hard drive copy, but I don’t know how long that takes).
Since I still had access to the file server from the iMac, I stopped at Best Buy to get a 4TB external hard drive (and some thumb drives for people that needed files right away via SneakerNet). Now I had the cloud backup *and* locally attached copies of everything in my office. I tracked down the original engineer that helped install and configure this setup and he helped me create a new share that is served directly from the domain controller. I copied over all of the backup files from the 4TB drive and tested connections. That worked, so I reviewed security settings with our GM and applied permissions to folders via AD security groups. Cloud backup was re-pointed to the new share and ran successfully from my iMac. I created cheat sheets on how to connect to the new share from both Mac and Windows clients and sent them out to our staff. Done, right? Well…
All of my Mac users had no problem connecting and seeing what they were permitted to see. Some Windows users were also completely fine, but others connected and could not view all of the folders they were allowed to see. I initially had access based enumeration turned on, so some hidden folders were expected (but not ones they should see). One colleague saw all folders in the office, but not over VPN from home (on a brand-new laptop). This TechNote pointed towards a local cache issue (which this article also talks about). We’ve been experimenting with various offline settings and most people are now connected successfully. Here’s how we are troubleshooting Windows connections now:
- Restart the workstation first, Roy
- Re-map the network drive using a different letter
- Use the full AD name (i.e. – domain.local\username)
- Delete local offline cache files
Dell/EMC sent me a notice this weekend that there is another update to the Unisphere OE (184.108.40.20699487), but I didn’t see anything relevant in the release notes. I did download the huge .gpg file anyways, but so far the health check is timing out and I can’t get it installed. Also forgot to mention that we rolled back two of the automatic Windows Server updates that installed in mid-February (and turned off automatic updates). That obviously isn’t a long-term strategy, so I’d like to get updated to Windows Server 2019 soon (VMware updates too). OneDrive, Teams, SharePoint and even Box/Dropbox for Business are all options that may come into play as well.
Serving files shouldn’t be rocket science. At least I felt a little better when the engineer said, “I’ve installed hundreds, if not a thousand, file servers like this and I’ve never seen one do what yours is doing.”
Trailblazing!Originally published by DK on March 1, 2021 at 5:12 pm
Experienced some weird technical glitches this week:
- The older of my two AWS instances just stopped running right after midnight one night. Uptime Robot notified me right away via email, but I didn’t see it until around 8am, so there was some significant downtime (relatively speaking). The instance restarted fine, but I don’t know what caused that (and Amazon didn’t notify me of any issues).
- I made one late night attempt earlier this week to move this site to my newer AWS instance, but had two issues: 1) my nginx virtual host config is apparently wrong, as it started sending requests to a different domain after I updated DNS and 2) creating a new Let’s Encrypt certificate failed (probably because the domain name didn’t match). Decided to just revert back for now and make another run at it later.
- The iOS WordPress app update this week seems to have now broken all functionality with this site. Before, I just had issues uploading featured images (they would upload to the media library successfully, but the app would never get a “finished” acknowledgement). Now it doesn’t even get that far. I’ll just post from macOS machines for now, but that’s a pain for mobile posting away from the house.
- My office workstation wasn’t set to automatically restart after a power outage, so it’s been off for a week. I have other machines to use at home, but that machine does my automated network share backups to the cloud, so I needed to get that issue resolved in person today. I went to check status when I returned home and initially couldn’t see it, but that turned out to be an issue with the DHCP lease expiring. All good now.
- Not a glitch, but decided to take a look at my Twitter settings tonight. Dropped some dead accounts (actually really surprised at how many accounts I follow that haven’t been active for 100 days or more), unmuted some people and started looking for a new profile picture. Twitter remains my last social media account – can’t decide if it stays or goes this year. Leaning towards keeping it, with a renewed focus on pruning and more active curation of what I mute and what I follow. Might go private too.
Thankful for a long weekend – stay warm, Minnesota peeps…Originally published by DK on February 12, 2021 at 9:09 pm
-5F and (probably dropping)Originally published by DK on February 12, 2021 at 11:26 am
Exhale. L.A., you’re on the clockOriginally published by DK on February 7, 2021 at 10:27 pm
Flexibility (and safety) is great, but nothing beats in personOriginally published by DK on January 21, 2021 at 11:02 pm
…but a more organized messOriginally published by DK on January 21, 2021 at 12:35 pm
These things are so finickyOriginally published by DK on January 9, 2021 at 6:46 pm
My head is still spinning after returning home from the last home game of a very strange 2020 season:
- Games with fewer people aren’t as stressful overall, but it still takes me several hours to wind down after the final whistle blows.
- Immensely thankful everyone on our team stayed healthy all season.
- I don’t walk nearly as many steps on game day since we got a Cushman for IT.
- Five regular seasons already? 2016: 5-3, 2017: 7-1, 2018: 5-3, 2019: 6-2, 2020: 3-5
- The people in the Vikings organization are extraordinary and it’s a pleasure to work with them year after year.
- Three stadiums are now newer than us in the NFL, but next year will feel like opening the building all over again.
Just a few things to clean up this week in the building, then on to 2021…Originally published by DK on December 20, 2020 at 6:14 pm
Once I found the right screwdriver, it wasn’t too bad (new battery on a Dell XPS 15)Originally published by DK on December 16, 2020 at 12:35 pm
More perks from having a kid at CandylandOriginally published by DK on December 15, 2020 at 4:06 pm
There will be wine tooOriginally published by DK on December 9, 2020 at 2:59 pm
Got my money’s worth on that oneOriginally published by DK on December 6, 2020 at 7:43 pm
First day @ CandylandOriginally published by DK on December 5, 2020 at 12:23 pm
Fans would probably be good tooOriginally published by DK on November 22, 2020 at 7:37 pm
Still seems like so much to doOriginally published by DK on November 20, 2020 at 9:31 pm
Remembering Sid 💜Originally published by DK on November 8, 2020 at 2:21 pm
Shouldn’t it be “more than” an hour?Originally published by DK on November 5, 2020 at 10:21 am
It’s coming back together again – slowlyOriginally published by DK on October 10, 2020 at 5:47 pm
Correctly, tooOriginally published by DK on September 14, 2020 at 1:13 pm
es • o • ter • ic
adj. Intended for or understood by only a small group, especially one with specialized knowledge or interests: synonym: mysterious.
Today was one of those days when I worked on so many strange things, I don’t know how I ever learned all this random stuff. I’m sure there are a million people that could step in and learn my job over time, but right now I don’t know what my colleagues will do if I get a fever and can’t go to work. I’m trying to diversify that risk through delegation, training and documentation, but there is a ton of specialization in our environment (which I’m sure is true for many other professions as well).
switchport access vlan 360
switchport mode access
power inline auto max 30000
service-policy output softmax
Who talks like that?Originally published by DK on September 11, 2020 at 11:21 pm
Are you ready for some football?Originally published by DK on September 9, 2020 at 7:31 am
I’ve been thinking a lot about safety lately. The pandemic, political and economic turmoil, return to work plans – all of these have safety components that give me anxiety and make 2020 feel like even more of a challenge. What exactly is the condition of being safe? I like these definitions: freedom from danger, risk, injury or loss; unhurt, harmless, cautious. Of course, safe can also mean unlikely to produce controversy or contradiction – also sometimes useful, but not what I’m writing about today.
The spread of COVID and the tools and policies available to manage the threat have become overtly political, which is unfortunate. I’m lucky that I really don’t have to fight any physical battles with people who won’t wear masks or follow guidelines and have the economic stability to isolate at home as much as I can during the pandemic. I’ve already decided I don’t *need* to visit casinos, go to movie theaters or drink in bars right now and I tend to favor take-out over eating inside at restaurants that are offering that option (and winter is coming, patio people). Correctly wearing a face mask to Target or the grocery store is literally the easiest thing you can do to keep the economy running, limit the spread of the virus and help keep people safe.
Outdoor activities are safer than indoor activities, which has resulted in a surge of bike sales and golf rounds. These are both things I love to do in a “regular” year, so this has been a great way to pass the time (and keep my move streak alive). It’s funny that some of the outdoor things I like to do aren’t always considered safe: danger is part of the appeal with downhill skiing and single-track mountain biking, for example. But even with those activities, I’m more cautious the older I get. Running has always been important to me, but I’ve even turned that down a notch, preferring strenuous hill walks to full out running for speed (at least for now).
I’m very fortunate that sports, concerts, travel and food have played significant roles in my life. Venue safety is part of my career, so I pay close attention to what others are doing. Unfortunately, we didn’t have a great experience with the Saints returning to live baseball in our neighborhood, so we only bought tickets to two games. Why? We didn’t feel safe: no security at the gates, seat maps that placed people at the drink rail immediately behind us, other fans not following the guidelines and limited staff to enforce the rules. I get the economics, but what is the long-term damage to your brand if people don’t feel safe? We did have a good experience at the Minnesota Zoo a few weeks back and expect to have a good experience at the Minnesota State Fair Food Parade next week, but it seems critical to me that you get safety figured out.
I miss live music, but I don’t see that industry coming back for a while. The next ticket I have is for a November 27th show at the Fitzgerald, a theater I love. Even with masks and a limited audience (which isn’t possible, since tickets are already sold), I can’t imagine any scenario that would make me feel safe enough to go on that date (and near-zero chance it doesn’t get rescheduled again like every other show I still have tickets for from 2020). I’ve also been reading a lot about Disney, and while thoughts of an uncrowded Disney World are tempting, my internal safety meter says tell Mickey we will see him again later.
The activities I’ve talked about so far mostly involve things that a majority of people would consider first-world privilege. I’m still employed, we’re not facing eviction and we have food on the table. Everyone in the family is still healthy. We have a bit of an economic safety net thanks to inheritance. We have access to reliable transportation. Middle class white males don’t have to face the same issues that women and people of color do as part of their daily existence (I’m working on a draft “On Race” post that I hope to publish soon). So what other safety issue is front and center for me? Crime rates.
When we moved from the suburbs to Lowertown, my mom was very concerned about our safety (even though she also lived in Saint Paul). The area has had ups and downs from a crime standpoint, but I’d never felt threatened here (even at night). The pandemic and the protests following the death of George Floyd have changed the look and atmosphere of downtown Saint Paul. According to the Pioneer Press, there have been more than forty recent arrests in the area around Mears Park and Union Depot for fighting, drug dealing and other criminal activity (including a shooting). Homelessness has also been on the rise and a small tent city can now be seen outside our windows near I-94. There are people needing obvious mental health assistance. I wish I knew the answers to these problems – what we are doing now isn’t working and something needs to change…Originally published by DK on August 23, 2020 at 9:07 pm
Lower Pentair lobby @ USBSOriginally published by DK on August 20, 2020 at 9:52 pm
The SSD drive upgrades do helpOriginally published by DK on August 19, 2020 at 1:35 pm
I forgot that was one of my first ever projects – NWA Newswire, February 11, 1992Originally published by DK on August 15, 2020 at 8:47 pm
Testing broadcast connectionsOriginally published by DK on July 28, 2020 at 3:41 pm
Too bad there won’t be a fourthOriginally published by DK on July 27, 2020 at 4:34 pm
Looked good today @ USBSOriginally published by DK on July 20, 2020 at 8:36 pm
Hoping it gets used in 2020Originally published by DK on June 17, 2020 at 1:28 pm
Troy Burne, Hudson, WisconsinOriginally published by DK on May 27, 2020 at 4:19 pm
Hope it sticks (even if it’s TV-only)Originally published by DK on May 7, 2020 at 9:15 pm
Still there, still closedOriginally published by DK on May 5, 2020 at 3:24 pm
Well-earned retirementOriginally published by DK on May 1, 2020 at 9:48 am
So bittersweet right nowOriginally published by DK on April 30, 2020 at 6:50 pm
Not too shabbyOriginally published by DK on April 1, 2020 at 12:43 pm
Man, I hope the Stones still get to playOriginally published by DK on March 16, 2020 at 7:29 pm
Colleen in the Allina command centerOriginally published by DK on March 15, 2020 at 3:59 pm
https://www.isitcanceledyet.com/Originally published by DK on March 11, 2020 at 5:06 pm
Foul ball at the very end of the sessionOriginally published by DK on March 10, 2020 at 3:22 pm
Baseball in this building looks so wonkyOriginally published by DK on March 6, 2020 at 3:29 pm
https://www.usbankstadium.com/connect-with-us/employmentOriginally published by DK on March 3, 2020 at 11:40 am
But what about Saint Paul?Originally published by DK on February 25, 2020 at 12:20 pm
Sunday, SUNDAY, SundayOriginally published by DK on February 15, 2020 at 9:28 pm
$20 snow cones for all!Originally published by DK on February 14, 2020 at 5:11 pm
Original turf sample from U.S. Bank StadiumOriginally published by DK on January 30, 2020 at 10:06 pm
Ich liebe dichOriginally published by DK on January 21, 2020 at 12:56 pm
Everybody is in your way, but it would be cool once or twiceOriginally published by DK on December 18, 2019 at 2:31 pm
Now get all the dust out of here, pleaseOriginally published by DK on December 15, 2019 at 9:41 am
And on Monday Night tooOriginally published by DK on December 8, 2019 at 4:46 pm
The Bean still rocks, thoughOriginally published by DK on December 6, 2019 at 1:58 pm
Thanks for the memories, MortensonOriginally published by DK on November 30, 2019 at 5:35 pm
Mood.Originally published by DK on November 19, 2019 at 2:23 pm