Category Archives: Cloud

Using AzureRM Powershell on a Mac

I’ve been watching with excitement ever since the announcement of powershell going open source and coming to multiple platforms like Mac and Linux.

My early experiments with it weren’t too productive – Powershell, with a base set of about 300 commandlets  worked, but all the add-in commandlets I needed to work with Azure did not.

Jessica Dean wrote an article and another about how to install the azurerm.netcore.preview modules with an earlier beta release of powershell for the mac.

Unfortunately, I could not get it to work – I suspect it required a specific alpha or beta version of powershell for mac, as I remember being a version ahead at the time it didn’t work.

Fast forward to today.

Powershell Mac Beta 4 has been available for a few weeks.

UPDATE: PowerShell Mac Beta 5 was released on August 2nd 2017. I’ve tested the steps in the article and they still work with Beta 5.

You can find the download for it here: https://github.com/PowerShell/PowerShell

Scroll down the page and look for the table with all the different installers, and grab the .pkg file. (note that if you hover over it, the filename will show the version:)

In the past I’ve installed the beta without issue, and I’m assuming if you just double click the installer, you’ll be fine.

I always start powershell by opening terminal (a built in mac app) and then typing powershell.

At this point if you type $(get-command).count it’ll return about 300 commandlets.

To install the Azure RM extensions, I did a search on the powershell gallery site: https://www.powershellgallery.com looking for azurerm.netcore.

I noticed that one of the search results didn’t have the .preview at the end, like the ones in Jessica’s article. And I noticed that the update date on it was from 3 days ago, this is a good sign!  (Update as of 8-10, Powershell Mac was updated to beta 5, but the AzureRM.NetCore module was still at 0.9.1 – I can confirm this works with beta 5 (though After installing beta 5, I did have to re-install AzureRM.Netcore)

 

 

 

 

 

 

 

I followed the instructions under install and got an error about not having permissions.

I exited powershell, then was brought back to terminal

Then from terminal I typed:

Sudo powershell

This brought me back into a powershell window, and I was able to run:

After that. I did a get-command and I now have nearly 900 commandlets!

Next I typed add-azurermaccount and I was able to get in!
(The auth system was a little weird, directing you to open a Url in a separate browser, sign in and copy/paste a code from powershell to the browser window, but it worked!)

This is great improvement!

Now, with proof that things are working, I wanted to confirm it worked without starting powershell with the sudo command.

Sure enough it works fine without it.

In closing, I’ve outlined the steps needed to install powershell and the azureRM extensions on a mac so you can work with azureRM powershell commands on your mac.

Let me know in the comments if this helps!

  • Jack

About Visual Studio Team Services Online

TFS online, otherwise known as VSTS, isn’t something I’ve had a lot of exposure to.

I thought I would jot down a few notes about it here..

VSTS sites all have a .visualstudio.com url – much like sharepoint online sites end in .sharepoint.com

Whereas SharePoint online is provisioned and managed via Office 365 and it’s control panel, VSTS is a bit different.

It’s provisioned in Azure, and as of this writing it’s a bit of a toss up if you’re better off using the old portal or the new – for me, the new one threw errors during provisioning, but the old ‘classic’ portal worked just fine.

Once it’s provisioned it shows up in both.

You can think of Visual Studio Team Services almost like a turnkey application you’d get from some of those web hosts that allow a single button wordpress install – you create it in Azure, but you manage it from itself, that is to say that once you’ve provisioned Visual Studio Team Services, and get a url like myVSTS.visualstudio.com, you’ll do all further configuration from that new URL – permissions, adding users, etc.

One area of confusion for me as an Azure administrator, was that I couldn’t see other Visual studio Team services in the new portal (they do show up in the old one), I also couldn’t set any meaningful permissions from either portal, making VSTS one of those services that the only way to manage it is for someone to give you rights to it after it is created.

Because of all this VSTS can be a bit of a “black box” at first – it doesn’t show up in azure like most azure services, yet it’s not managed from the office 365 portal like most of office 365.

What really helped me was to just create my own VSTS so I could see the UI, play around with it and see what it’s about.

If you do that and want to delete it later, this is also a bit perplexing. You might think you can delete it from azure since that’s where you created it, but you instead unlink an account from azure, then go to the url you were given (ie myVSTS.visualstudio.com) and go into settings and Delete the account from there.

Here’s the official article on how to delete an unneeded VSTS: https://www.visualstudio.com/en-us/docs/setup-admin/team-services/delete-or-recover-your-account-vs

  • Jack

Steps to take when visualstudio online will not load

I had an odd thing happen today – I was invited to several different visualstudio online sites.

lets call them

abc.visualstudio.com and xyz.visualstudio.com

I was able to get to one of them just fine, and the other would not load no matter what I did – I even tried a different browser on a different PC, no luck.

I opened a case with microsoft and they suggested the following:

Regarding your issue: Please try the steps below

  1. Go to aka.ms/vssignout
  2. Clear all cookies
  3. Close browser
  4. Reopen browser in private mode
  5. Try to log in

This seemed to fix things, though I can’t explain why.

The reason I can’t explain it is, I had this issue in 3 configurations:

  • Chrome on mac
  • Safari on mac
  • Chrome on windows

I followed Microsoft’s steps on the Mac, using Safari, and was able to get right in afterwards.

Next I tried Chrome on windows without following the microsoft steps, and that worked too.

Logic would have it that I would need to repeat those steps the windows box before that worked, but that wasn’t the case, no idea why, but thought the steps should be posted for future reference just in case.

Powershell script to remove duplicate, old modules

If you’ve installed modules in powershell such as the AzureRM collection, and have updated to newer versions of said modules, it’s possible that you might have both the old and new versions installed. This can slow down some commands in powershell (get-command for example)

I wrote a small script to remove old versions of Powershell modules using what I found at https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/14934876-update-module-needs-flag-to-remove-previous-versio) as a starter.

Note that when I say “Remove old versions” I mean it leaves the most current one you have and removes the older ones – if your system is 3 versions behind, it will not go fetch new commands first (though it does remind you of the command to do that)

Hope this helps!

Beware! Changes to AzureRM powershell commandlets may break your online scripts!

This week I ran into an interesting problem:

A powershell commandlet failed when run on Azure Automation Services, but it worked for me on my local machine.

It was a simple command

if you’re not familiar with it, it brings back a list of all the subscriptions you have access to in Azure.

The problem is, it used to return properties named “SubscriptionName” and “SubscriptionID”

These were recently changed to “Name” and “ID” causing code that used the old properties to fail.

This brings to light the importance of keeping your azureRM modules up to date on your local workstation, so you’re testing the same thing. In my case, I was using version 3.8, but 4.1 was current.

My last post listed a few commands of interest for keeping up with installed modules, as a recap, here are a few of interest:

One problem I ran into today: After updating all my Modules, powershell was slow as can be.  I had installed updated versions, but hadn’t removed the old ones.

Maybe there is a better way, and if so, please comment below, but I ended up creating a small script to remove prior versions. For findability, that’ll be in my next post.

 

Some misc powershell commands for working with azure PS modules

This is one of those posts I’m doing mostly for myself to keep things in one place.

Remove an old version of a module:

$Latest = Get-InstalledModule (modulename); Get-InstalledModule (modulename) -AllVersions | ? {$_.Version -ne $Latest.Version} | Uninstall-Module -WhatIf

another way to do the same:

get-installedmodule modname -requiredversion  x.x.x | uninstall-module

(taken from https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/14934876-update-module-needs-flag-to-remove-previous-versio)

Show what modules are installed:

get-installedmodule

Show what versions of a specific module are installed

get-installedmodule -name modname -allversions

 

Update the azureRM module:

update-module azurerm -force

Install over a stubborn module:

install-module azurerm.cognitiveservices -force

 

 

 

Using a Mac in a Microsoft world: SQLPro from Hankinsoft

While most of this blog covers my professional experience with SharePoint running on the windows platform, I’ve also spent a bunch of time as a Mac user.

In the past, I’ve almost always remoted into windows machines to get work done – SharePoint, Powershell, SQL Management Studio, etc.. all required this approach.

This was pretty much the only option when things were on premise but now that the cloud is such a focus, there was one tool I was particularly interested in:

A tool for interacting with Microsoft SQL Server in Azure from my mac.

I tried a handful of options including some open source cross platform tools, some paid cross platform tools and a mac only gem called SQLPRO for MSSQL.

The free options were kinda 1990’s java ugly. One wouldn’t even connect, and one connected but I ran into problems enumerating fields.

Of the paid options, the best bang for the buck for me was Hankinsoft’s SQLPro for MSSQL.

sqlpromssql

This tool is a native mac application (not a java port) and it shows: the app is fast, responsive, and looks like a mac app, supporting newer features such as the mac’s special full screen mode.

I had no trouble connecting to PaaS SQL in Azure, and was able to do all the things I’d normally want to do. The tool has code completion,  the ability to drag and drop field and table names from the navigation pane on the left to the editor, some nice features like the ability to auto indent sql code, and the ability to run multiple queries (separated by semicolons;) and display the results from each one in the results window, just like MS SQL Management Studio does!

SQLPro is a pretty stable product, and that’s no surprise, as the developer has been making database tools for quite some time now. (There are versions for MySQL, SQLite, and Postgres, as well as a ‘studio’ version that supports multiple types of databases)

SQLPro was $79.99 as of this writing – This is a one time, buy it, it’s yours price, and it’s very competitive vs other tools that also offer the ability to connect to and work with Microsoft SQL server.

So far I’m really enjoying being able to run queries against SQL in Azure natively on a mac – This is a great tool and highly recommended!

SQLPro for MSSQL for Mac
by Hankinsoft Development

Modify-Sublicense PowerShell function for modifying Office 365 Sublicenses

Modify-Sublicense

An example call would be:

Modify-Sublicense -upn "joe@yourcompany.com" -PrimaryLicense "TENANT:ENTERPRISEPACK -SublicensesToRemove @("SWAY","YAMMER_ENTERPRISE") -SublicensesToAdd @("SHAREPOINTENTERPRISE", "SHAREPOINTWAC")

If you’re new to powershell, scroll down past the code below for some additional tips.

Naturally, sublicenses you had before will REMAIN unless you’ve removed them.
AND
Sublicenses you DID NOT HAVE before will NOT BE ADDED unless you specifically add them.

This is important because of how the licensing works, which has been covered in other blog posts my myself and others.

I’ve used the function below as part of a larger script with good results.

As always, TEST BEFORE YOU USE IT!

New to powershell?

Here are a few tips:

  • Since the code is a function, you’ll need to copy-paste it to a script of your own before you can use it
  • In powershell, functions need to appear in your script above/before you use them.
  • There are ways to load the function in memory, so you can call it as if it was a native command. See How to add functions to your powershell session
  • An array in powershell can have zero or 1 or many items, if you need to pass a single value, just pass it as an array with one value. That would look like this: @("Value1", "Value2")

– Jack

Replace an expiring Client Secret in an app for SPO

What are we even calling these things these days? Apps for SharePoint Online? Apps for Office 365?

This article is about the apps we build using the new app model that was introduced along with SharePoint 2013. It’s the main way of developing functionality for SharePoint Online.

The apps are hosted in Azure (or on the hosting provider of your choosing)

When these Apps are created/installed a Client secret is used to ensure that communication between your externally hosted app and SharePoint Online is secure and not coming from an attacker.

Unfortunately these certificates expire.

Ours have.

The article below talks about replacing them.

https://msdn.microsoft.com/en-us/library/office/dn726681(v=office.15).aspx

We also opened a ticket with Microsoft Premier Support which revealed a few more tidbits.

  • It takes like 24 hours for the new certificate to propogate through your system, leaving your app out of commission for at least that long if you don’t renew before it expires.
  • The article above mentions, but does not give an example of, extending the date from the default 1 year to 3 years. I’ve copied some of the correspondence with Mustaq Patel from Microsoft, who helped us through the process (Thanks Mustaq!)

Note for the scripts below, you’ll need your clientID this is in the web.config of your website that’s hosted in Azure.  As luck would have it, the person at our company who would have had this info was on vacation. Since it’s in the web.config of the running app, it made sense to just pull the actual web.config in use. I did this via FTP, using the steps in this article to configure an FTP account to gain access to the server:

https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure/

Update: You can also find the clientID by going to any sharepoint site that uses the app, Site Settings->Site App Permissions.

It’ll be the guid between the last pipe symbol and the @ symbol

Example i:oi.t|blahblahblah|abcdef-1234-this-is-aguid-and-is-what-you-want@7a534-no-the-guid-you-want-123

(thanks to Mustaq for pointing this out!)

  1. Connect to MSOnline using tenant admin user with below powershell in SharePoint 2013 powershell
  2. Get ServicePrincipals and keys. Printing $keys will give 3 records, replace each KeyId in key1, key2 and key3. You can also see EndDate of each key. Confirm if your expired key shows there. Also note that clientId needs to match as per your clientId.

  3. Generate new ClientSecret for this clientID. Please note it uses clientId set in #2. Also ClientSecret is valid for 3 years.

  4. Copy the output of $newClientSecret.

  5. Replace the Web.config with this ClientId and ClientSecret. Please note we don’t need SecondaryClientSecret appsettings.

  6. Wait for 24 hours to propagate ClientSecret to SPO

FINALLY! It’s now possible to migrate data to SharePoint Online without data loss.

After 18 months of persistence, we’re finally able to migrate to SPO.

Up until yesterday, it wasn’t possible to migrate user data from SharePoint if the user was missing from Active Directory. It is now if you’re migrating to SharePoint online and using the new migration API!

 

Early on we ran into what was a pretty glaring problem for us, and I suspect for anyone else trying to migrate using a client side migration tool.

The Problem:

One of the data types in SharePoint is the “user” data type.
This is most commonly used/seen in document libraries – it shows you who last edited/updated a document.
It’s also commonly used as a field type in a SharePoint list.
For example, you might have a sharepoint list named “legal cases”

The problem is, using the client side API used for most migration tasks involving SharePoint Online, you can’t insert a name of a person if that name can’t be found in Azure Active Directory.

The 18 month Path to Resolution:

While it didn’t take Microsoft 18 months of actual work to fix this, it was about 18 months from start to finish.

I won’t bore you with the details, but the take away here is to never give up. We were told ‘no’ or ‘it can’t be done’ or ‘that’s by design’ multiple times.  This was an important issue so we pressed on, reaching out to every contact we knew, from multiple levels within our organization to multiple levels within Microsoft. All that persistence paid of!

Sidenote

One conversation along the way that was particularly interesting was a special presentation Microsoft had arranged where we got to talk to the person who led Microsoft’s internal migrations from on premise to office 365. I was eager to ask them how they had moved list data with abandoned user data – I was certain they had some internal tool, or did some back door load that wasn’t available to outsiders, or maybe had some script that identified every instance of the missing users and recreated them in AD so the migration could complete.  When I asked, they skirted the issue. I pressed on. After they told me, I understood why. They didn’t do it. When pressed they said they had to leave some data on premise because there was no way to move this data to SPO. On that day, I felt both validated and let down at the same time!

The Solution:

At ignite 2015, Microsoft announced a new Migration API for SharePoint Online.

On the same day, ShareGate announced that it would support this new API with it’s new ‘insane mode’. I spoke with a few people, some thought this new API would resolve the issue, while others said it would not – it didn’t at that time. 🙁

Shortly after our case must have reached one of the senior escalation engineers at Microsoft – I remember being told that the new API resolved the issue, then going back with evidence that it didn’t and I think that’s when traction really picked up. We supplied a Business impact statement and Microsoft added the fix to the list of things they were working on.  The feedback I got down the road was that this ended up being a huge undertaking for them. It wasn’t nearly as easy as one would think, due to how SharePoint is structured internally.  There were multiple setbacks, but we received excellent communication and updates. The time line didn’t bother me – I was thrilled to know it was being worked on.

Fast forward to fairly recently and we received word that the fix was approved and would be moved into our tenant. This was great news! Work didn’t end there however.

Microsoft went into depth about the work done to fix this issue.

While I had expected some new API, or an option when sending data like “override if missing” no such changes were needed. Microsoft updated the migration API to handle all of the needed back end stuff seamlessly. They did not update the CSOM. This meant that for this to work, the new API had to be used.

We were already using ShareGate coupled with insane mode which uses the API. I remember from past conversation with them that ShareGate uses a combination of insane mode and CSOM – even when insane mode is being used – I figured this would mean ShareGate would need to be aware of the changes to the migration API and would need to handle things differently. For example, in the past, ShareGate could replace a missing user with a user of our choice, this would no longer be necessary.

ShareGate was great to work with – they had long been aware of the user data migration issue and understood what I was talking about almost immediately.  Once the API had been updated, the three of us worked together to ensure that Sharegate’s test tenant also had the new Mirgration API updates so they could code up a solution.

I’m soo pleased to say that Sharegate turned this around in about a month, and we received a beta last Friday.  Even more impressive is that the very first beta from ShareGate with support for this worked like a charm!

Even more impressive is that the very first beta from ShareGate with support for this worked like a charm!

Sharegate has released version 5.8 with this functionality Today (2/29/2016).

Confirmed: Microsoft has already rolled this out to all O365 tenants. (as of 2/29/2016)

Special kudos and thanks to Brent Vezzoso and the unnamed hero’s within Microsoft who worked so tirelessly to make this happen. I can’t tell you enough how much this means to me, my company, the SharePoint community, and to Microsoft. You’ve done a great thing here!

  • Jack