Category Archives: Cloud

FINALLY! It’s now possible to migrate data to SharePoint Online without data loss.

After 18 months of persistence, we’re finally able to migrate to SPO.

Up until yesterday, it wasn’t possible to migrate user data from SharePoint if the user was missing from Active Directory. It is now if you’re migrating to SharePoint online and using the new migration API!

 

Early on we ran into what was a pretty glaring problem for us, and I suspect for anyone else trying to migrate using a client side migration tool.

The Problem:

One of the data types in SharePoint is the “user” data type.
This is most commonly used/seen in document libraries – it shows you who last edited/updated a document.
It’s also commonly used as a field type in a SharePoint list.
For example, you might have a sharepoint list named “legal cases”

The problem is, using the client side API used for most migration tasks involving SharePoint Online, you can’t insert a name of a person if that name can’t be found in Azure Active Directory.

The 18 month Path to Resolution:

While it didn’t take Microsoft 18 months of actual work to fix this, it was about 18 months from start to finish.

I won’t bore you with the details, but the take away here is to never give up. We were told ‘no’ or ‘it can’t be done’ or ‘that’s by design’ multiple times.  This was an important issue so we pressed on, reaching out to every contact we knew, from multiple levels within our organization to multiple levels within Microsoft. All that persistence paid of!

Sidenote

One conversation along the way that was particularly interesting was a special presentation Microsoft had arranged where we got to talk to the person who led Microsoft’s internal migrations from on premise to office 365. I was eager to ask them how they had moved list data with abandoned user data – I was certain they had some internal tool, or did some back door load that wasn’t available to outsiders, or maybe had some script that identified every instance of the missing users and recreated them in AD so the migration could complete.  When I asked, they skirted the issue. I pressed on. After they told me, I understood why. They didn’t do it. When pressed they said they had to leave some data on premise because there was no way to move this data to SPO. On that day, I felt both validated and let down at the same time!

The Solution:

At ignite 2015, Microsoft announced a new Migration API for SharePoint Online.

On the same day, ShareGate announced that it would support this new API with it’s new ‘insane mode’. I spoke with a few people, some thought this new API would resolve the issue, while others said it would not – it didn’t at that time. 🙁

Shortly after our case must have reached one of the senior escalation engineers at Microsoft – I remember being told that the new API resolved the issue, then going back with evidence that it didn’t and I think that’s when traction really picked up. We supplied a Business impact statement and Microsoft added the fix to the list of things they were working on.  The feedback I got down the road was that this ended up being a huge undertaking for them. It wasn’t nearly as easy as one would think, due to how SharePoint is structured internally.  There were multiple setbacks, but we received excellent communication and updates. The time line didn’t bother me – I was thrilled to know it was being worked on.

Fast forward to fairly recently and we received word that the fix was approved and would be moved into our tenant. This was great news! Work didn’t end there however.

Microsoft went into depth about the work done to fix this issue.

While I had expected some new API, or an option when sending data like “override if missing” no such changes were needed. Microsoft updated the migration API to handle all of the needed back end stuff seamlessly. They did not update the CSOM. This meant that for this to work, the new API had to be used.

We were already using ShareGate coupled with insane mode which uses the API. I remember from past conversation with them that ShareGate uses a combination of insane mode and CSOM – even when insane mode is being used – I figured this would mean ShareGate would need to be aware of the changes to the migration API and would need to handle things differently. For example, in the past, ShareGate could replace a missing user with a user of our choice, this would no longer be necessary.

ShareGate was great to work with – they had long been aware of the user data migration issue and understood what I was talking about almost immediately.  Once the API had been updated, the three of us worked together to ensure that Sharegate’s test tenant also had the new Mirgration API updates so they could code up a solution.

I’m soo pleased to say that Sharegate turned this around in about a month, and we received a beta last Friday.  Even more impressive is that the very first beta from ShareGate with support for this worked like a charm!

Even more impressive is that the very first beta from ShareGate with support for this worked like a charm!

Sharegate has released version 5.8 with this functionality Today (2/29/2016).

Confirmed: Microsoft has already rolled this out to all O365 tenants. (as of 2/29/2016)

Special kudos and thanks to Brent Vezzoso and the unnamed hero’s within Microsoft who worked so tirelessly to make this happen. I can’t tell you enough how much this means to me, my company, the SharePoint community, and to Microsoft. You’ve done a great thing here!

  • Jack

An o365 Support case that was handled so quickly, I had to write about it!

I was so impressed with how fast this was turned around, I had to write about it…

A special shout out to Microsoft Sr. Escalation Engineer Sridhar Narra and the support team including Linda PorcelliLoren Jacobson and Tiffany Evenson who worked to resolve this issue so quickly!

Apparently in email addresses, the single quote is a legal character.

So names like: Bill.O’Brian@yourcompany.com are legal in the email sense.

Addresses like this can cause havoc on programmers.

Well it turns out that we had a few people like this in our company and this did not play well with Office 365 at all.

The first, and easiest thing we noticed was that if you found a person like Bill O’Brian in search and tried to click on them, the profile page for the user threw an error.

We opened a case with Microsoft and I was expecting something short like “Don’t use quotes in your email” but instead, they looked into and fixed it. FAST.

I’m used to changes in o365 taking 3 months to make it from ticket to production, this one took a few weeks, from the time we opened the ticket, to the time the issue was resolved in production.

This is a pace of improvement and change I’ve not seen until now and it’s really exiting to see things get fixed this quickly!

A special shout out to Microsoft Sr. Escalation Engineer Sridhar Narra and the support team including Linda Porcelli, Loren Jacobson and Tiffany Evenson who worked to resolve this issue so quickly!

As a side note, a  quotation mark in an email address has issues in other places of SharePoint. Our case was split into several cases. Sridhar’s team worked on the profile page, There’s another team looking at another case which I’ll update in the future when I have more information.  If your organization has issues similar to this, please open a premier ticket.

– Jack

 

A quick update on where I’ve been…

I noticed today that I haven’t posted anything in a while here- notably the entire month of August. That’s partly due to vacation schedules in the summer – I went on a family vacation then went to Laracon for 4 days in Kentucky to learn more about PHP the programming language that powers more websites in the world than any other language. (Like 81%!)

Investing in On premise SharePoint today feels like putting new tires on a car you’re going to junk in 2 months.

Another reason though, is that as we move to office 365, my job role is changing. It’s been a good year now since I’ve built up a new on premise farm for work, and even the announcement of SharePoint 2016 on-premise doesn’t really excite me.  It appears that on-premise has a limited future, and the cloud is the direction of the future, not just for SharePoint but for lots of technologies.  Investing in On premise SharePoint today feels like putting new tires on a car you’re going to junk in 2 months.

So as I work more with o365 and less with on premise, the issues I face are different.  In August I created a series of 20 or so SPO training videos and put them up in the Office 365 video portal (that I’ll be talking about in my talk at SPSTC in October) That took a good part of my free time.

I’ve spent a lot of time opening issues with Microsoft. Today I have a success story to share, which I’ll post separately on. I believe I have another success story coming in Late October that’s HUGE for the SharePoint community, and critical for the migration to o365 from on premise.

As always, thanks for stopping by!

 

Script for Auto-Adjusting Site quotas in SharePoint Online / O365 based on current useage

** IMPORTANT UPDATE #2 -this is no longer needed.

As of September 2015, Site Quotas are no longer needed!

The screenshot below came from our tenant settings screen – Set it to Automatic and forget about site quotas!

AutoStorageSPOnline

** IMPORTANT UPDATE RELATED TO SHAREPOINT / OFFICE 365 VIDEO SITES ** (Updated April 29th 2015)

If you use the script below, you may want to put in a filter to prevent the script from working on any site with the managed path of /portals/

Explanation:

Microsoft recently added Video portals to office 365.

Ours was added this morning.

I looked at it in the admin screen because I was curious what site collections had been created.

I noticed something peculiar: The Storage Quota was 0.

2015-04-29_17-34-23

You can’t set a quota to zero. Not in the UI, not in PowerShell.

Well tonight my script ran and guess what, it reset my quote on the Video portal.

Originally, Video portal storage was supposed to come out of the overall SPOnline allotment, so this may not be a big deal, maybe we needed to set it anyways?

But, since it’s not possible to set zero manually, I wonder if this was one site you didn’t have to manage/pre-allocate storage too? Or maybe MS decided to make storage unlimited?

I’m not really sure, but in the short term, I thought I’d best warn people that there are some unknowns here as it relates to using the below script with Office Video.

 

AND NOW BACK TO OUR ORIGINAL ARTICLE, as it was published on April 15th:

The year is 2015, You’ve just been given “TONS” of storage on SharePoint online, and someone says “Lets give everyone a 200GB site quota”.

Seems like a great idea, you’ll never run out of space, so why not set the limit high?

Well, it turns out, the word “Quota” has a different meaning in SharePoint Online / Office 365 than it did in SP on Premise.

In the On Premise version of SharePoint, the quota was a limit.
In SharePoint Online, it’s an Allocation.

What’s the difference you ask?

Say you have a 1000GB of storage on SharePoint on premise.

With On premise SharePoint, you can allocate a 200gb quota to 10 sites, “over committing” what you actually have. It works because space isn’t ‘reserved’ for that site, it’s just a limit. You’re telling on premise sharepoint “Don’t let any site get bigger than 200gb”.

Take a similar situation on SharePoint Online:

Say you have 1000GB of storage on SPO

You can only allocate 200GB quotas 5 times – each time you do, your total available drops by 200GB so by the 5th one, you have nothing left to give. This is true, even if the sites are empty!

So SharePoint online works a little differently, at least in 2015 it does – maybe one day this article won’t be relevant, but it is today.

What are we to do if we want to give users basically unlimited sized site collections

Now the question: What are we to do if we want to give users basically unlimited sized site collections, but we can’t allocate large numbers to EACH site collection?

Well, here’s what I did – I wrote a script that looks at how much storage each site collection is using, then adjusts it so there is a certain amount of ‘headroom’.

I run the script daily via a scheduled task.

I also have another script that sets up the connection to SharePoint Online which runs first, if you need that part, it’s elsewhere on this site.

The logic is fairly straightforward, but lets do an example:

All sites should be 4GB or more over the size used:

  • An empty site would have 4gb allocated
  • A site using 3gb would have 7gb allocated
  • A site using 10gb would have 14gb allocated.

Make sense?

That logic is pretty simple, take the size of the site, add 4gb to it, that’s it’s allocation.

For performance reasons, it would be great if we weren’t constantly adjusting each and every site, every time we run the script, the $slack setting helps with that.

Here’s the script:

(If you need help scheduling a task, I have a blog post and video here about that. )

– Jack

Script for assigning SharePoint Licenses to Office365

Adding SharePoint licenses to Office 365 can be a bit tricky.

If you add the E3 license, you get EVERYTHING that comes with E3, if that’s what you need, great, but what if you ONLY want SharePoint, and not Lync, email, etc??

I ran into this recently and used a few resources to come up with a script.

This article was really helpful:  http://www.powershellmagazine.com/2012/04/23/provisioning-and-licensing-office-365-accounts-with-powershell/

As was some script work by an awesome guy I work with named Chris.

The tricky thing here is you can’t directly grant just a SharePoint license in MSOL E3…

You have to do it subtractively.

Let me explain…..

Say you have 3 letters, A, B & C

You might expect to add a license for b like this:

Add-license -option B

It doesn’t work that way. (At least not in 2015 when I wrote this)

Instead you have to say:

Add-License -disable A C

No problem you say.

“I’ll just add code to disable A C”

That’s great, until….

Microsoft adds Option D

Now, when you try

Add-License -disable A C

You’ve just assigned a B and D license, when you only wanted to assign a B license.

Now you see the issue….

The solution is not too hard – we can pull a list of all options available, remove the one we want, and then build the disable list from that.

This way we won’t get caught when Microsoft springs options EFGHI on us.

The full script is below.

Note: there are some unused functions in the script for setting a password – if you have brand new users to Office 365, they may never have used their identity and will need the password reset and sent to them, if that’s the case, just add the call to Reset-Password -upn $upn at the appropriate place(s)

Here’s the script:

 

Create a ShareGate User mapping file between on Premise AD and o365 / Azure AD

We use ShareGate to migrate content.

We recently started using ShareGate to migrate content from On Premise to SharePoint Online.

When I did this, I found that one of our domain’s users kept showing up as errors in ShareGate – it said it could not find the user in SharePoint Online.

ShareGate has a nice feature for mapping users from one system to users in another – but doing this manually to any scale would be pretty time consuming.

Thankfully, ShareGate lets us save the mappings, which are just XML files with a .sgum file extension.

Wouldn’t it be great if there was a way to automate creating a mapping file like this for everyone in the domain at once?

Have a look at the script below, it pulls all the user accounts from an OU in AD, then looks up each user to find them in MSOL (Office 365 Azure AD) Then grabs the o365 display name and makes the mapping . Any user not found is logged so it can be dealt with separately.

The whole thing is written out as a complete .sgum file, ready to import into ShareGate the next time you migrate!

Note I didn’t figure out the XML stuff in a vacuum – I found an article on powershellmagazine.com to be very helpful and noted it in the script.

– Jack

Is ‘SharePoint Online’ becoming ‘Office 365 Sites’?

There’s been a bit of chatter for some time now about the future of SharePoint.
An article over a year ago portrayed SharePoint as a technology Stack, much like .net

I couldn’t help but notice in my office 365 site, that SharePoint “Sites” aren’t actually branded “SharePoint” in the panel now:
Office365Portal_Dec_12_2014

See that?
Every other branded product has its brand name.
We see “Excel Online” not “Spreadsheet”
OneDrive is there.
So is Delve

And both PowerPoint Online and and OneNote Online use two lines to describe what those do.
In fact, there are exactly the same number of characters in the word “PowerPoint” as there are in “SharePoint”

Lets look at one more thing…
See that tile with the “V” – that’s not Visio, it’s Video, you know, Office 365 Video:

Office365VideoTitle

Now, if “Video” by itself means “Office 365 Video”, does it follow suit that “Sites” by itself means “Office 365 Sites”?

Lets look at a few more things in the o365 interface.
When I clicked on “Sites” the first time, I got this screen:
Office365Sites_screen1
Now here we can relax, Microsoft hasn’t rolled Sites into OneDrive. If you note the url, you’ll see it’s going to a mysite location. It’s the first time I’ve used this tenant, and mysites have never been setup for the user that I’m using.

After the mysite was setup, I went back to the first screenshot, and again clicked on the “Sites” link.
This time I came here:
Office365Sites_screen2
Now lets take a moment and examine that screen shall we?

  • There’s no mention of SharePoint…
  • Those familiar with the SharePoint 2013 branding might recognize the “S>” logo.
  • Those familiar with SharePoint infrastructure might notice that the URL is a SharePoint MySite.

If that’s a mysite, maybe the “team site” is different?

Curious, I had to create a standard old “team site” and see what it looked like – I did this in the SharePoint online admin panel (yes that’s still there, under the hood) Note, that I did this before the above screenshot was taken, which is why you see “Public Site” and “Team Site” in the screenshot.

Ok so what does a “Team Site” look like?
Office365Sites_TeamSite

See the word SharePoint anywhere on that screenshot?
I didn’t think so.
I even brought down the settings menu to see if there was any mention of it. (no)

Times they are a changing. If I didn’t know any better, I’d say that the SharePoint Brand Name is on it’s way out.
Is it time to rethink the “SharePointJack” brand as well?

Automating SharePoint Deployments in Windows Azure using PowerShell

There’s a nice article on MSDN titled “Automating SharePoint Deployments in Windows Azure using PowerShell

The article talks about building up a sharepoint environment in Azure. The main difference between this and setting up an office 365 instance that I wrote about in my last post, is that in Azure, you’re in complete control over all the VM’s that make up your environment. If you have MSDN you get a monthly spending allowance on Azure, so you should be able to play around with this stuff without incurring any costs (So long as you don’t leave everything running 24×7) – Seems perfect for test environments. At the bottom of the article are links to a few articles that discuss using PowerShell with Azure.