Salı, 29 Mayıs 2018 / Published in Uncategorized

In my last blog post Adding Cross-Cutting Memory Caching to an HttpClientFactory in ASP.NET Core with Polly I actually failed to complete my mission. I talked to a few people (thanks Dylan and Damian and friends) and I think my initial goal may have been wrong.

I thought I wanted "magic add this policy and get free caching" for HttpClients that come out of the new .NET Core 2.1 HttpClientFactory, but first, nothing is free, and second, everything (in computer science) is layered. Am I caching the right thing at the right layer?

The good thing that come out of explorations and discussions like this is Better Software. Given that I’m running Previews/Daily Builds of both .NET Core 2.1 (in preview as of the time of this writing) and Polly (always under active development) I realize I’m on some kind of cutting edge. The bad news (and it’s not really bad) is that everything I want to do is possible it’s just not always easy. For example, a lot of "hooking up" happens when one make a C# Extension Method and adds in into the ASP.NET Middleware Pipeline with "services.AddSomeStuffThatIsTediousButUseful()."

Polly and ASP.NET Core are insanely configurable, but I’m personally interested in the 80% or even the 90% case. The 10% will definitely require you/me to learn more about the internals of the system, while the 90% will ideally be abstracted away from the average developer (me).

I’ve had a Skype with Dylan from Polly and he’s been updating the excellent Polly docs as we walk around how caching should work in an HttpClientFactory world. Fantastic stuff, go read it. I’ll steal some here:

ASPNET Core 2.1 – What is HttpClient factory?

From ASPNET Core 2.1, Polly integrates with IHttpClientFactory. HttpClient factory is a factory that simplifies the management and usage of HttpClient in four ways. It:

  • allows you to name and configure logical HttpClients. For instance, you may configure a client that is pre-configured to access the github API;

  • manages the lifetime of HttpClientMessageHandlers to avoid some of the pitfalls associated with managing HttpClient yourself (the dont-dispose-it-too-often but also dont-use-only-a-singleton aspects);

  • provides configurable logging (via ILogger) for all requests and responses performed by clients created with the factory;

  • provides a simple API for adding middleware to outgoing calls, be that for logging, authorisation, service discovery, or resilience with Polly.

The Microsoft early announcement speaks more to these topics, and Steve Gordon‘s pair of blog posts (1; 2) are also an excellent read for deeper background and some great worked examples.

Polly and Polly policies work great with ASP.NET Core 2.1 and integrated nicely. I’m sure it will integrate even more conveniently with a few smart Extension Methods to abstract away the hard parts so we can fall into the "pit of success."

Caching with Polly and HttpClient

Here’s where it gets interesting. To me. Or, you, I suppose, Dear Reader, if you made it this far into a blog post (and sentence) with too many commas.

This is a salient and important point:

Polly is generic (not tied to Http requests)

Now, this is where I got in trouble:

Caching with Polly CachePolicy in a DelegatingHandler caches at the HttpResponseMessage level

I ended up caching an HttpResponseMessage…but it has a "stream" inside it at HttpResponseMessage.Content. It’s meant to be read once. Not cached. I wasn’t caching a string, or some JSON, or some deserialized JSON objects, I ended up caching what’s (effectively) an ephemeral one-time object and then de-serializing it every time. I mean, it’s cached, but why am I paying the deserialization cost on every Page View?

The Programmer’s Hindsight: This is such a classic programming/coding experience. Yesterday this was opaque and confusing. I didn’t understand what was happening or why it was happening. Today – with The Programmer’s Hindsight – I know exactly where I went wrong and why. Like, how did I ever think this was gonna work? 😉

As Dylan from Polly so wisely points out:

It may be more appropriate to cache at a level higher-up. For example, cache the results of stream-reading and deserializing to the local type your app uses. Which, ironically, I was already doing in my original code. It just felt heavy. Too much caching and too little business. I am trying to refactor it away and make it more generic!

This is my "ShowDatabase" (just a JSON file) that wraps my httpClient

public class ShowDatabase : IShowDatabase
{
    private readonly IMemoryCache _cache;
    private readonly ILogger _logger;
    private SimpleCastClient _client;
 
    public ShowDatabase(IMemoryCache memoryCache,
            ILogger<ShowDatabase> logger,
            SimpleCastClient client)
    {
        _client = client;
        _logger = logger;
        _cache = memoryCache;
    }
 
    static SemaphoreSlim semaphoreSlim = new SemaphoreSlim(1);
  
    public async Task<List<Show>> GetShows()
    {
        Func<Show, bool> whereClause = c => c.PublishedAt < DateTime.UtcNow;
 
        var cacheKey = "showsList";
        List<Show> shows = null;
 
        //CHECK and BAIL - optimistic
        if (_cache.TryGetValue(cacheKey, out shows))
        {
            _logger.LogDebug($"Cache HIT: Found {cacheKey}");
            return shows.Where(whereClause).ToList();
        }
 
        await semaphoreSlim.WaitAsync();
        try
        {
            //RARE BUT NEEDED DOUBLE PARANOID CHECK - pessimistic
            if (_cache.TryGetValue(cacheKey, out shows))
            {
                _logger.LogDebug($"Amazing Speed Cache HIT: Found {cacheKey}");
                return shows.Where(whereClause).ToList();
            }
 
            _logger.LogWarning($"Cache MISS: Loading new shows");
            shows = await _client.GetShows();
            _logger.LogWarning($"Cache MISS: Loaded {shows.Count} shows");
            _logger.LogWarning($"Cache MISS: Loaded {shows.Where(whereClause).ToList().Count} PUBLISHED shows");
 
            var cacheExpirationOptions = new MemoryCacheEntryOptions();
            cacheExpirationOptions.AbsoluteExpiration = DateTime.Now.AddHours(4);
            cacheExpirationOptions.Priority = CacheItemPriority.Normal;
 
            _cache.Set(cacheKey, shows, cacheExpirationOptions);
            return shows.Where(whereClause).ToList(); ;
        }
        catch (Exception e)
        {
            _logger.LogCritical("Error getting episodes!");
            _logger.LogCritical(e.ToString());
            _logger.LogCritical(e?.InnerException?.ToString());
            throw;
        }
        finally
        {
            semaphoreSlim.Release();
        }
    }
}
 
public interface IShowDatabase
{
    Task<List<Show>> GetShows();
}

I’ll move a bunch of this into some generic helpers for myself, or I’ll use Akavache, or I’ll try another Polly Cache Policy implemented farther up the call stack! Thanks for reading my ramblings!

UPDATE: Be sure to read the comments below AND my response in Part 2.

Sponsor: SparkPost’s cloud email APIs and C# library make it easy for you to add email messaging to your .NET applications and help ensure your messages reach your user’s inbox on time. Get a free developer account and a head start on your integration today!

© 2018 Scott Hanselman. All rights reserved.





Salı, 29 Mayıs 2018 / Published in Uncategorized

I’m continuing to upgrade my podcast site https://www.hanselminutes.com to .NET Core 2.1 running ASP.NET Core 2.1. I’m using Razor Pages having converted my old Web Matrix Site (like 8 years old) and it’s gone very smoothly. I’ve got a ton of blog posts queued up as I’m learning a ton. I’ve added Unit Testing for the Razor Pages as well as more complete Integration Testing for checking things "from the outside" like URL redirects.

My podcast has recently switched away from a custom database over to using SimpleCast and their REST API for the back end. There’s a number of ways to abstract that API away as well as the HttpClient that will ultimately make the call to the SimpleCast backend. I am a fan of the Refit library for typed REST Clients and there are ways to integrate these two things but for now I’m going to use the new HttpClientFactory introduced in ASP.NET Core 2.1 by itself.

Next I’ll look at implementing a Polly Handler for resilience policies to be used like Retry, WaitAndRetry, and CircuitBreaker, etc. (I blogged about Polly in 2015 – you should check it out) as it’s just way too useful to not use.

HttpClient Factory lets you preconfigure named HttpClients with base addresses and default headers so you can just ask for them later by name.

public void ConfigureServices(IServiceCollection services)
{
    services.AddHttpClient("SomeCustomAPI", client =>
    {
        client.BaseAddress = new Uri("https://someapiurl/");
        client.DefaultRequestHeaders.Add("Accept", "application/json");
        client.DefaultRequestHeaders.Add("User-Agent", "MyCustomUserAgent");
    });
    services.AddMvc();
}

Then later you ask for it and you’ve got less to worry about.

using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;

namespace MyApp.Controllers
{
    public class HomeController : Controller
    {
        private readonly IHttpClientFactory _httpClientFactory;

        public HomeController(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }

        public Task<IActionResult> Index()
        {
            var client = _httpClientFactory.CreateClient("SomeCustomAPI");
            return Ok(await client.GetStringAsync("/api"));
        }
    }
}

I prefer a TypedClient and I just add it by type in Startup.cs…just like above except:

services.AddHttpClient<SimpleCastClient>();

Note that I could put the BaseAddress in multiple places depending on if I’m calling my own API, a 3rd party, or some dev/test/staging version. I could also pull it from config:

services.AddHttpClient<SimpleCastClient>(client => client.BaseAddress = new Uri(Configuration["SimpleCastServiceUri"]));

Again, I’ll look at ways to make this even simpler AND more robust (it has no retries, etc) with Polly soon.

public class SimpleCastClient
{
    private HttpClient _client;
    private ILogger<SimpleCastClient> _logger;
    private readonly string _apiKey;

    public SimpleCastClient(HttpClient client, ILogger<SimpleCastClient> logger, IConfiguration config)
    {
        _client = client;
        _client.BaseAddress = new Uri($"https://api.simplecast.com"); //Could also be set in Startup.cs
        _logger = logger;
        _apiKey = config["SimpleCastAPIKey"]; 
    }

    public async Task<List<Show>> GetShows()
    {
        try
        {
            var episodesUrl = new Uri($"/v1/podcasts/shownum/episodes.json?api_key={_apiKey}", UriKind.Relative);
            _logger.LogWarning($"HttpClient: Loading {episodesUrl}");
            var res = await _client.GetAsync(episodesUrl);
            res.EnsureSuccessStatusCode();
            return await res.Content.ReadAsAsync<List<Show>>();
        }
        catch (HttpRequestException ex)
        {
            _logger.LogError($"An error occurred connecting to SimpleCast API {ex.ToString()}");
            throw;
        }
    }
}

Once I have the client I can use it from another layer, or just inject it with [FromServices] whenever I have a method that needs one:

public class IndexModel : PageModel
{
    public async Task OnGetAsync([FromServices]SimpleCastClient client)
    {
        var shows = await client.GetShows();
    }
}

Or in the constructor:

public class IndexModel : PageModel
{
    private SimpleCastClient _client;

    public IndexModel(SimpleCastClient Client)
    {
        _client = Client;
    }
    public async Task OnGetAsync()
    {
        var shows = await _client.GetShows();
    }
}

Another nice side effect is that HttpClients that are created from the HttpClientFactory give me free logging:

info: System.Net.Http.ShowsClient.LogicalHandler[100]
      Start processing HTTP request GET https://api.simplecast.com/v1/podcasts/shownum/episodes.json?api_key=
System.Net.Http.ShowsClient.LogicalHandler:Information: Start processing HTTP request GET https://api.simplecast.com/v1/podcasts/shownum/episodes.json?api_key=
info: System.Net.Http.ShowsClient.ClientHandler[100]
      Sending HTTP request GET https://api.simplecast.com/v1/podcasts/shownum/episodes.json?api_key=
System.Net.Http.ShowsClient.ClientHandler:Information: Sending HTTP request GET https://api.simplecast.com/v1/podcasts/shownum/episodes.json?api_key=
info: System.Net.Http.ShowsClient.ClientHandler[101]
      Received HTTP response after 882.8487ms - OK
System.Net.Http.ShowsClient.ClientHandler:Information: Received HTTP response after 882.8487ms - OK
info: System.Net.Http.ShowsClient.LogicalHandler[101]
      End processing HTTP request after 895.3685ms - OK
System.Net.Http.ShowsClient.LogicalHandler:Information: End processing HTTP request after 895.3685ms - OK

It was super easy to move my existing code over to this model, and I’ll keep simplifying AND adding other features as I learn more.

Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!

© 2018 Scott Hanselman. All rights reserved.





Salı, 29 Mayıs 2018 / Published in Uncategorized

Last week while upgrading my podcast site to ASP.NET Core 2.1 and .NET. Core 2.1 I moved my Http Client instances over to be created by the new HttpClientFactory. Now I have a single central place where my HttpClient objects are created and managed, and I can set policies as I like on each named client.

It really can’t be overstated how useful a resilience framework for .NET Core like Polly is.

Take some code like this that calls a backend REST API:

public class SimpleCastClient
{
    private HttpClient _client;
    private ILogger<SimpleCastClient> _logger;
    private readonly string _apiKey;

    public SimpleCastClient(HttpClient client, ILogger<SimpleCastClient> logger, IConfiguration config)
    {
        _client = client;
        _client.BaseAddress = new Uri($"https://api.simplecast.com");
        _logger = logger;
        _apiKey = config["SimpleCastAPIKey"];
    }

    public async Task<List<Show>> GetShows()
    {
        var episodesUrl = new Uri($"/v1/podcasts/shownum/episodes.json?api_key={_apiKey}", UriKind.Relative);
        var res = await _client.GetAsync(episodesUrl);
        return await res.Content.ReadAsAsync<List<Show>>();
    }
}

Now consider what it takes to add things like

  • Retry n times – maybe it’s a network blip
  • Circuit-breaker – Try a few times but stop so you don’t overload the system.
  • Timeout – Try, but give up after n seconds/minutes
  • Cache – You asked before!
    • I’m going to do a separate blog post on this because I wrote a WHOLE caching system and I may be able to "refactor via subtraction."

If I want features like Retry and Timeout, I could end up littering my code. OR, I could put it in a base class and build a series of HttpClient utilities. However, I don’t think I should have to do those things because while they are behaviors, they are really cross-cutting policies. I’d like a central way to manage HttpClient policy!

Enter Polly. Polly is an OSS library with a lovely Microsoft.Extensions.Http.Polly package that you can use to combine the goodness of Polly with ASP.NET Core 2.1.

As Dylan from the Polly Project says:

HttpClientFactory in ASPNET Core 2.1 provides a way to pre-configure instances of HttpClient which apply Polly policies to every outgoing call.

I just went into my Startup.cs and changed this

services.AddHttpClient<SimpleCastClient>();

to this (after adding "using Polly;" as a namespace)

services.AddHttpClient<SimpleCastClient>().
    AddTransientHttpErrorPolicy(policyBuilder => policyBuilder.RetryAsync(2));

and now I’ve got Retries. Change it to this:

services.AddHttpClient<SimpleCastClient>().
    AddTransientHttpErrorPolicy(policyBuilder => policyBuilder.CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 2,
            durationOfBreak: TimeSpan.FromMinutes(1)
    ));

And now I’ve got CircuitBreaker where it backs off for a minute if it’s broken (hit a handled fault) twice!

I like AddTransientHttpErrorPolicy because it automatically handles Http5xx’s and Http408s as well as the occasional System.Net.Http.HttpRequestException. I can have as many named or typed HttpClients as I like and they can have all kinds of specific policies with VERY sophisticated behaviors. If those behaviors aren’t actual Business Logic ™ then why not get them out of your code?

Go read up on Polly at https://githaub.com/App-vNext/Polly and check out the extensive samples at https://github.com/App-vNext/Polly-Samples/tree/master/PollyTestClient/Samples.

Even though it works great with ASP.NET Core 2.1 (best, IMHO) you can use Polly with .NET 4, .NET 4.5, or anything that’s compliant with .NET Standard 1.1.

Gotchas

A few things to remember. If you are POSTing to an endpoint and applying retries, you want that operation to be idempotent.

"From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result."

But everyone’s API is different. What would happen if you applied a Polly Retry Policy to an HttpClient and it POSTed twice? Is that backend behavior compatible with your policies? Know what the behavior you expect is and plan for it. You may want to have a GET policy and a post one and use different HttpClients. Just be conscious.

Next, think about Timeouts. HttpClient’s have a Timeout which is "all tries overall timeout" while a TimeoutPolicy inside a Retry is "timeout per try." Again, be aware.

Thanks to Dylan Reisenberger for his help on this post, along with Joel Hulen! Also read more about HttpClientFactory on Steve Gordon’s blog and learn more about HttpClientFactory and Polly on the Polly project site.

Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!

© 2018 Scott Hanselman. All rights reserved.





Salı, 29 Mayıs 2018 / Published in Uncategorized

In my last post I talked about Caching and some of the stuff I’ve been doing to cache the results of a VERY expensive call to the backend that hosts my podcast.

As always, the comments are better than the post! Thanks to you, Dear Reader.

The code is below. Note that the MemoryCache is a singleton, but within the process. It is not (yet) a DistributedCache. Also note that Caching is Complex(tm) and that thousands of pages have been written about caching by smart people. This is a blog post as part of a series, so use your head and do your research. Don’t take anyone’s word for it.

Bill Kempf had an excellent comment on that post. Thanks Bill! He said:

The SemaphoreSlim is a bad idea. This "mutex" has visibility different from the state it’s trying to protect. You may get away with it here if this is the only code that accesses that particular key in the cache, but work or not, it’s a bad practice. As suggested, GetOrCreate (or more appropriate for this use case, GetOrCreateAsync) should handle the synchronization for you.

My first reaction was, "bad idea?! Nonsense!" It took me a minute to parse his words and absorb. Ok, it took a few hours of background processing plus I had lunch.

Again, here’s the code in question. I’ve removed logging for brevity. I’m also deeply not interested in your emotional investment in my brackets/braces style. It changes with my mood. 😉

public class ShowDatabase : IShowDatabase
{
    private readonly IMemoryCache _cache;
    private readonly ILogger _logger;
    private SimpleCastClient _client;
  
    public ShowDatabase(IMemoryCache memoryCache,
            ILogger<ShowDatabase> logger,
            SimpleCastClient client){
        _client = client;
        _logger = logger;
        _cache = memoryCache;
    }
  
    static SemaphoreSlim semaphoreSlim = new SemaphoreSlim(1);
   
    public async Task<List<Show>> GetShows() {
        Func<Show, bool> whereClause = c => c.PublishedAt < DateTime.UtcNow;
  
        var cacheKey = "showsList";
        List<Show> shows = null;
  
        //CHECK and BAIL - optimistic
        if (_cache.TryGetValue(cacheKey, out shows))
        {
            return shows.Where(whereClause).ToList();
        }
  
        await semaphoreSlim.WaitAsync();
        try
        {
            //RARE BUT NEEDED DOUBLE PARANOID CHECK - pessimistic
            if (_cache.TryGetValue(cacheKey, out shows))
            {
                return shows.Where(whereClause).ToList();
            }
  
            shows = await _client.GetShows();
  
            var cacheExpirationOptions = new MemoryCacheEntryOptions();
            cacheExpirationOptions.AbsoluteExpiration = DateTime.Now.AddHours(4);
            cacheExpirationOptions.Priority = CacheItemPriority.Normal;
  
            _cache.Set(cacheKey, shows, cacheExpirationOptions);
            return shows.Where(whereClause).ToList(); ;
        }
        catch (Exception e) {
            throw;
        }
        finally {
            semaphoreSlim.Release();
        }
    }
}
  
public interface IShowDatabase {
    Task<List<Show>> GetShows();
}

SemaphoreSlim IS very useful. From the docs:

The System.Threading.Semaphore class represents a named (systemwide) or local semaphore. It is a thin wrapper around the Win32 semaphore object. Win32 semaphores are counting semaphores, which can be used to control access to a pool of resources.

The SemaphoreSlim class represents a lightweight, fast semaphore that can be used for waiting within a single process when wait times are expected to be very short. SemaphoreSlim relies as much as possible on synchronization primitives provided by the common language runtime (CLR). However, it also provides lazily initialized, kernel-based wait handles as necessary to support waiting on multiple semaphores. SemaphoreSlim also supports the use of cancellation tokens, but it does not support named semaphores or the use of a wait handle for synchronization.

And my use of a Semaphore here is correct…for some definitions of the word "correct." 😉 Back to Bill’s wise words:

You may get away with it here if this is the only code that accesses that particular key in the cache, but work or not, it’s a bad practice.

Ah! In this case, my cacheKey is "showsList" and I’m "protecting" it with a lock and double-check. That lock/check is fine and appropriate HOWEVER I have no guarantee (other than I wrote the whole app) that some other thread is also accessing the same IMemoryCache (remember, process-scoped singleton) at the same time. It’s protected only within this function!

Here’s where it gets even more interesting.

  • I could make my own IMemoryCache, wrap things up, and then protect inside with my own TryGetValues…but then I’m back to checking/doublechecking etc.
  • However, while I could lock/protect on a key…what about the semantics of other cached values that may depend on my key. There are none, but you could see a world where there are.

Yes, we are getting close to making our own implementation of Redis here, but bear with me. You have to know when to stop and say it’s correct enough for this site or project BUT as Bill and the commenters point out, you also have to be Eyes Wide Open about the limitations and gotchas so they don’t bite you as your app expands!

The suggestion was made to use the GetOrCreateAsync() extension method for MemoryCache. Bill and other commenters said:

As suggested, GetOrCreate (or more appropriate for this use case, GetOrCreateAsync) should handle the synchronization for you.

Sadly, it doesn’t work that way. There’s no guarantee (via locking like I was doing) that the factory method (the thing that populates the cache) won’t get called twice. That is, someone could TryGetValue, get nothing, and continue on, while another thread is already in line to call the factory again.

public static async Task<TItem> GetOrCreateAsync<TItem>(this IMemoryCache cache, object key, Func<ICacheEntry, Task<TItem>> factory)
{
    if (!cache.TryGetValue(key, out object result))
    {
        var entry = cache.CreateEntry(key);
        result = await factory(entry);
        entry.SetValue(result);
        // need to manually call dispose instead of having a using
        // in case the factory passed in throws, in which case we
        // do not want to add the entry to the cache
        entry.Dispose();
    }

    return (TItem)result;
}

Is this the end of the world? Not at all. Again, what is your project’s definition of correct? Computer science correct? Guaranteed to always work correct? Spec correct? Mostly works and doesn’t crash all the time correct?

Do I want to:

  • Actively and aggressively avoid making my expensive backend call at the risk of in fact having another part of the app make that call anyway?
  • What I am doing with my cacheKey is clearly not a "best practice" although it works today.
  • Accept that my backend call could happen twice in short succession and the last caller’s thread would ultimately populate the cache.
  • My code would become a dozen lines simpler, have no process-wide locking, but also work adequately. However, it would be naïve caching at best. Even ConcurrentDictionary has no guarantees"it is always possible for one thread to retrieve a value, and another thread to immediately update the collection by giving the same key a new value."

What a fun discussion. What are your thoughts?

Sponsor: SparkPost’s cloud email APIs and C# library make it easy for you to add email messaging to your .NET applications and help ensure your messages reach your user’s inbox on time. Get a free developer account and a head start on your integration today!

© 2018 Scott Hanselman. All rights reserved.





Salı, 29 Mayıs 2018 / Published in Uncategorized

Reading Time: 12 minutes

What is one of the deadliest traps we can fall into in user interface design? How can we avoid it? What is the difference between simple and dumb user interface? How can we make interfaces more intuitive? Have you ever asked any of these questions? What are some good practices for user interface design? This article may finally give you the answers you have been looking for.

The problem with user interface design

It should be always part of the user interface design process to ask one simple question. And, to ask this simple question repeatedly. Is this really intuitive? And, we should put an extra emphasis on the “really”. However, there is a significant difference between what we, designers, think is intuitive and what is “really” intuitive. Meaning, is it intuitive for the user?

Understanding and seeing this difference can be very difficult for three reasons. First, we have library of knowledge about user interface design. I know what you think. Well, this should be a good thing don’t you think? Well, yes and know. Knowledge of user interface design helps us do our job, create great user interfaces. Unfortunately, it can also do us harm. How?

The burden of knowledge

Simply said, our knowledge of user interface design changes the way we think and see. Let me give you a simple example. When designer opens a magazine, there are two types of things she will see. On one hand, she will see what everyone else sees, unless she has some perception related health condition. She will see text, images and whatever is presented on the pages of the magazine she is reading.

On the other hand, she will also see something different, something only designer can see. She may “see” the grid, patterns used in the layout of the page, visual hierarchy, and so on. Sure, everyone reading the magazine will see these things as well. However, she is more likely to pay attention to them. In the end, she spent countless hours studying user interface design and its examples.

These hours of study and practice made her the designer she is today. However, these hours altered her perception. Her knowledge allows her see subtle nuances and see them faster. This is a good thing as well as bad thing. It makes her vulnerable to make assumptions that may not be true. For example, about the degree to which the design is intuitive. Yet, this is only her assumption.

When users design for users

Knowledge is not the only problem. The second problem, also potentially leading to assumptions, is her own experience as a user. We all used some user interface, in some shape or form. And, we used this experience to form certain assumption about how that interface worked so we could use it. Again, this is a good thing as well as bad thing. Otherwise, we would have to always learn it again.

As you probably realized, all the experiences we gained as users can have the same impact as our knowledge. Well, I should say our professional knowledge, to distinguish between those two because experience itself is also form of knowledge. Anyway, our own experience can also make us vulnerable to make assumptions that may not be true, and change how intuitive something seems to us.

Thanks to our experience and learned patterns, we can see something as intuitive while others may not. A simple test. Do you have a smartphone? It is easy to use, right? Find someone who is used to a “dumb” phone. Then, let that person try your smartphone and watch what will happen. You will probably see that the phone you considered easy to use can quickly become a puzzle for someone else.

It was only your own assumption, built on your experiences, that your smartphone is easy to use. With time, you learned how to use it, the correct patterns. As a result, something unfamiliar became almost a part of your body. These patterns are not something we are born with. We learned them through our experiences. And, we are constantly learning some new when we use something new.

The more you use it …

Finally, the third problem. The time. The longer we work on some user interface design, and the more we use it, the more familiar it will become to us. This can, in turn, again distort our perception about how intuitive the user interface really is. How intuitive something is is basically just a result of a use repeated over amount of time. Thanks to this, anything can be intuitive.

A seemingly hard solution for a seemingly simple problem

So, what is the way out of this trap? We have to start asking that one simple question. Is this really intuitive? Then, before we answer this question, we have to remind ourselves about the three problems we just discussed. One, our professional knowledge changes our perception. Two, our own experiences change our perception. And, third the time we use the interface changes our perception.

The result? We are might make assumptions that may not true. As you can see, the solution for this problem is simple and hard, both at the same time. It requires us to regularly “check” ourselves to see whether we are making assumptions based on our knowledge or experience. It requires us to keep in mind that what may seem intuitive users interface design may not be intuitive for others, for those people who will use the interface in the future.

In addition to keeping this, in a sense, a beginner’s mind, we should also test our answers. Meaning, find a few people who never saw that interface and let them try it. This is probably the most objective way to ensure we are as unbiased as we can be. So, let’s make it a habit to regularly ask ourselves, is this really intuitive? Then, let’s always doubt, question and test our answers.

Simple, but not dumb

When we talk about intuitive user interface design, we like to talk about interfaces that often excel at simplicity. These interfaces usually use a lot of white space. And they also limit the amount of content present at the same time. As a result, the interface becomes uncluttered, clean and visually pleasing. We can say that it looks simple. However, that doesn’t mean it is dumb down.

Some people think that this is the key to intuitive user interface design, to make the interface dumber. False assumption appearing real. There is a difference between dumb and simple interface. It is similar to the difference between person who is dumb and person who explains concept in a simple language. In the first case, you will learn nothing. In the second, you can learn a lot.

So, let’s not confuse these two terms, simple and dumb. How can we define a simple user interface? I think that a simple user interface is an interface that helps the user achieve her goal in the easiest way possible. This means that process of user interface design has to always with the user in mind, so to speak. We have to always ask what the main user’s goal or job is.

Then, when we find the answer, we can start working on our main goal. That is creating an interface that will help the user achieve her goal in the easiest, fastest and least demanding way possible. This means putting together all we now about user interface design as well as examples of existing interfaces. Then, we should remove all that is not essential, or that could make it harder for the user to achieve her goal.

I know that this is a very general, or vague. And, it is not helpful. Every user interface is different. Therefore, every process of user interface design will have to be at least a bit different as well. This process has to be customized for each and every situation. However, there are some practices we can apply in the majority of use cases.

One thing we have to remember is the problems with intuitive user interface design we discussed above. What we think is user’s main goal or job may not be true. It is only our own assumption. We have to always talk to people who match the profile of our target user. Otherwise, we may base our user interface design process on wrong job and end up in a wrong destination with unusable product.

A few tips to make the user interfaces more intuitive

We discussed the problem with intuitive user interface design and the way to solve it. Then, we quickly talked about what is a simple user interface and how created a usable definition for it. Now, let’s dive one degree deeper. Let’s take a look at some tips and practices that will help us make the result of our user interface design process really intuitive.

Make it digestible

One way to make user interface more simple and easier to use is by using something called progressive disclosure. Put simply, we should present the user only with the most important information or options at first, say, on the main screen. This can be some basic instructions or introduction. Then, when user really wants to learn more, we should offer her ways to do so.

This will help the user start using the interface without being overwhelmed and paralyzed by amount of information, options or features. We have to remember that our primary goal, at least in the beginning, is to help the user start using the interface. Our goal is not teaching her everything there is so she can achieve mastery in using the interface, or become a superuser.

We want to help the user adopt the interface and learn just what is necessary to interact with the interface and get the job done. So, if you have some information that the user has to know, create a very short summary and start with that. The user doesn’t need to know every feature and option available. After that, provide the user with some way to learn more when she wants.

What if we have interface that has to present the user with a large amount of information? In that case, we can still use progressive disclosure and make the information more digestible. We can split the information into smaller blocks and show them to the user one by one. This way, we can also control in what order user gets the information and decrease the chance of skipping something important.

We can apply the same practice everywhere. If the interface requires the user to provide some information, we can ask only for the most important. For example, we can ask only for email and password, the data that might be really necessary to start using the interface. Then, when user logs in for the first time, we can ask her for additional information. Again, in small digestible chunks.

Use affordances

Another way to make any result of user interface design intuitive is by using elements that look like one can interact with them. This sounds simple. However, if we take a look current at the trend of ghost buttons and even brutalism, there are some examples where it is hard to distinguish what is a button. You have two squares and one is a button, probably the one with barely readable label.

We can have intuitive user interface with elements that look almost the same. User has to have some chance to distinguish between what is interactive and what is not. This doesn’t mean that we should never use things such as ghost buttons. It means that we must be sure that button looks like some user can interact with, that it is clickable. And, this applies to all elements. Obvious always wins.

So, yes, use your creativity in your user interface design process. Just don’t go too far. Test your ideas with real people, real users. Don’t build the whole thing on your own unfounded assumptions. Always seek feedback.

Provide feedback

When you ask user to do some action, provide her with some feedback after the action is completed. Intuitive user interface design should always aim to create a dialog between the user and the interface. It should be a two-way street. When user do something, the interface should respond. The same if something has to be done. The interface should notify the user and show the steps she has to make.

Interaction with interface should feel like interaction with living person. The interface should always respond to user’s action in some way. Intuitive user interface should create a dialog with the user. This conversation can be in any shape and form. It can be through text notifications, animations, sounds, etc. When it comes to feedback, there are two things we have to keep in mind.

First, we have to make sure that we will not overwhelm the user. Showing 10 or 20 notifications will not help the user. It will only annoy and overwhelm her. Imagine having a conversation with someone and being flooded with constant flow of question. The same here. Respond to the user with notifications, but keep the amount low. Always consider alternative ways to respond to the user.

Second, when there is something that needs to be done, we should show the user how to do it. It is not enough to say this or that has to be done or there is some problem. Intuitive user interface design is about showing the user not only the information. It is also about showing her the steps so she can act on it immediately. Don’t assume that she knows what to do, show it step by step.

Speak users’ language

Another way to make the interface intuitive is by using the same language as the user. This doesn’t mean just using English if our users are English speakers. It also means using the words they are likely to use, and in the way they are using them. If we know that our target user group uses some slang or variation of language we can use it too and customize the user interface design.

Using a concrete slang or language variation is helpful for several reasons. First, the interface will be more familiar to the user. Since we are speaking the same language, literally, it will be easier for the user to understand the interface and start using it. The interface will no longer be a stranger. It will be like talking with someone from the same city, country, etc.

Second, this also means that we can use smaller amount of information to help the user understand how the interface works. It will be easier for us to explain how features of the interface work and how to use them. There will be also less space for potential misunderstanding. Finally, user will more likely use the interface because, thanks to the same language, she will feel more comfortable.

Using the same language doesn’t end with words. We can also use icons that are familiar to our target user group. This will also make the communication and interaction between users and the interface faster and easier. The same applies to other elements. If we know that our target user is used to seeing some elements we should use them. The result will be again greater familiarity.

As before, we have to make sure we know our target user group very well. Using the same language can help us make user interface design more intuitive. However, this is based on the assumption that we know our target group well. If not, we will “achieve” the exact opposite result. The interface will be hard to use for people and the mutual communication will be broken from the very beginning.

Make the user interface efficient

This will be the last practice for more intuitive user interface design. As we discussed above, the goal should be creating an interface that helps the user achieve her goal in the easiest and fastest way possible. There should be no inefficient or unnecessary interactions. If some action can be done in just two steps, we should not force the user to make three.

The same applies to information and using progress disclosure. We should provide the user with information that is important and relevant for the current situation. If there is some additional information, we should provide the user with a simple way to get to this information. However, we should not force the user to read it if it is not necessary or important in that situation.

We should also provide the user with some default options or choices she can use. This will make the interaction and communication between her and the interface faster and easier. It will also help the user give us the right type of information because we will give her an example of what we are asking for. This can be useful for designing more intuitive forms contained in the interface.

Finally, we should reiterate to never overuse notifications and messages and overwhelm the user. Yes, it is important to provide the user with feedback. However, there are different types of feedback as well as priorities for each. Notification should not be a default type of feedback. We should use it only when there is something important the user should know about.

Otherwise, we should consider other, and more subtle, ways to notify the user feedback that something is happening in the background. The goal of user interface design process should be creating an interface that is unobtrusive. An interface that can become invisible.

Closing thoughts on user interface design

This is all I have for you today guys. I hope you enjoyed this article. I also hope that you learned something that changed the way you think about interface design. There is one final message I would like you to remember. User interface design starts and ends with users. It is not about creating something that will look good on Dribbble or Behance or even your portfolio.

User interface design is about creating interfaces that help users achieve their goals and finish specific jobs quickly and easily. The goal of user interface design should be creating a tool that not looks good, but that is also pleasure to use. Aesthetics are not everything. Design is also about function. Design also has to work. With that, thank you for your time and have a great day!

Do you have any questions, recommendations, thoughts, advice or tip you would like to share with other readers of this blog, and me? Great! Please share it in a comment. Or, if you want to keep things more "private", feel free to contact me on twitter or send me a mail. I would love to hear from you.

Did you like this article? Please subscribe.

Are you on social media? Let’s connect! You can find me on Twitter and Dribbble.

Salı, 29 Mayıs 2018 / Published in Uncategorized

Image copyright Jeff Bashton

Image caption
From the "point of view of someone who can’t see" his bank’s upgrade is "appalling", says Jeff

HSBC, Metro Bank and Halifax have all admitted to failings after redesigning websites that made it hard for their blind or visually impaired customers to access full services online.

When Jeff Bashton needs to transfer cash or pay a bill, he logs on to his bank’s website, clicks a few buttons and the job is done. It’s quick, easy and efficient. Or at least it used to be – before the bank, HSBC, upgraded its website, with elaborate headings, mortgage offers and banners advertising insurance or foreign currency.

"HSBC, in their infinite wisdom, decided to design an all-singing, all-dancing website," he tells BBC Radio 4’s Money Box programme.

"They think they’ve done a good job," he says, but "from the point of view of someone who can’t see, it’s an appalling job".

New obstacles

It might look good – but all the extra text on the site makes it almost impossible for Jeff, who is blind, to do his banking on his own.

Image copyright Getty Images

Jeff uses a screen reader – software with a synthesised voice that reads aloud what’s on the screen. So he can find headings, click on links and read bank statements. But the pages must be carefully designed with screen readers in mind.

And if there’s too much information scattered throughout, it means basic headings like "transfer money" or "make a payment" get lost in the website’s clutter.

Jeff is frustrated that what used to be a simple process has now become unreliable. He’s been forced to do telephone banking, which he says takes him longer.

"In the past, I would have given HSBC eight or nine out of 10. Now I wouldn’t give it higher than two or three," he laments.

Media playback is unsupported on your device

Media captionHow do screen readers work?

Jeff is not alone. Andy Godfrey, who is also blind, has been struggling to find his bank statements online since HSBC made the changes. Yet when he told the bank about the problem, they replied: "Oh, don’t worry, we’ll send you a video". He has now changed banks.

In a statement, HSBC told Money Box it has just finished upgrading the site again, and saying "we apologise to those who may have experienced problems accessing the site during this update". Jeff Bashton says now he detects a slight improvement.

No prompts

But HSBC has not been the only bank to cause problems for its blind customers. Metro Bank recently altered its online login page. So instead of one box to type in a full password, the new system introduced three separate boxes, each requiring a single character from the password.

Image copyright Darren Paskell

Image caption
His bank’s website upgrades locked Darren Paskell out of his bank account

Because there was no information announcing the changes that the screen reader could pick up, the new process totally stumped Metro Bank customer, Darren Paskell.

"I was typing my whole password in the first box and then going to the login button and obviously it was rejecting it because it was invalid data," he says.

It meant he was locked out of his account, unable to log in or manage his finances independently. It was something Mr Paskell says he found insulting and disempowering.

"It was basically set up to allow me to fail, again and again and again," he asserts.

Image copyright PA

Image caption
Metro Bank says its update was not tested "with sufficient accessibility technology"

Darren, who works in IT, spent half an hour on the phone trying to make Metro Bank understand the issue, without success. Eventually, he used his own technical expertise to work out a way of logging into his account.

Metro Bank said: "We have reviewed Mr Paskell’s case and would like to apologise for letting him down. The new update wasn’t tested with sufficient accessibility technology, meaning screen readers were not able to interpret all fields on the login page."

The bank says it is committed to getting it fixed.

Legal requirement

More than two million people in the UK have vision problems that can’t be corrected with prescription glasses. And over a third of a million are registered blind or partially sighted.

Under the Equality Act, firms must not discriminate against anyone by failing to provide them with a service. And that includes banking services. Companies that don’t comply could face a claim for damages.

Image caption
"If you’re blind and can’t access your banking online then you risk handing over a lot of control to others," says the BBC’s Lee Kumutat

In February, Halifax bank altered its website leaving customers like Richard Godfrey-McKay (who is blind, like his brother Andy) unable to read his bank statements online. It took the bank more than two months to rectify the problem and it has left Richard feeling like a second-class citizen.

"If it had been sighted people who couldn’t access the website, I’m bloomin’ sure that Halifax would have done something about it, pretty darn quick. That ‘delay’ is totally unacceptable.

"Banks have an anticipatory duty, to ensure that their website offering is accessible to its blind and partially-sighted customers."

Halifax acknowledges there has been a problem and says it was fixed quickly.

UK Finance, which represents the banks, says: "The industry is committed to ensuring services are accessible to everyone and any temporary issues that do arise are resolved as soon as possible."

Early consultation

One bank that claims it’s getting things right for blind customers is Barclays. According to its head of digital banking, Paul Smyth, the company involves disabled people right from the start, as part of its development and testing process.

Including them early, he says, makes it easier and cheaper – as illustrated three years ago, when the bank redesigned its mobile banking app.

Image copyright PA

Image caption
Including accessibility from the start, Barclays says, is much cheaper than correcting mistakes later on

"We were aware of a number of disabled customer issues with it," Paul Smyth explains.

"To fix those at the time would have cost hundreds of thousands of pounds. But by considering accessibility right from the start and involving disabled customers, it was a fraction – a tenth – of the cost."

So it seems banks can make new and complex websites accessible – if they approach it in the right way.

Salı, 29 Mayıs 2018 / Published in Uncategorized

Introduction

In computer science, the Dining Philosophers problem is an example problem often used in concurrent algorithm design to illustrate synchronization issues and techniques for resolving them.The Dining Philosophers problem was originally formulated in 1965 by Edsger Dijkstra as a student exam exercise and was soon thereafter put into its current formulation by Tony Hoare. Wikipedia describes the problem as follows:

“Five silent philosophers sit at a round table with bowls of spaghetti. Forks are placed between each pair of adjacent philosophers.

The Philosophers' TableEach philosopher must alternately think and eat. However, a philosopher can only eat spaghetti when they have both left and right forks. Each fork can be held by only one philosopher and so a philosopher can use the fork only if it is not being used by another philosopher. After an individual philosopher finishes eating, they need to put down both forks so that the forks become available to others. A philosopher can take the fork on their right or the one on their left as they become available, but cannot start eating before getting both forks.

Eating is not limited by the remaining amounts of spaghetti or stomach space; an infinite supply and an infinite demand are assumed.

The problem is how to design a discipline of behavior (a concurrent algorithm) such that no philosopher will starve; i.e., each can forever continue to alternate between eating and thinking, assuming that no philosopher can know when others may want to eat or think.”

Before rushing off to code your own solution, consider solving the Dining Philosophers problem using the Concurrency Explorer (ConcX), a free, open source tool from the Avian Computing Project. ConcX is an interactive tool designed to help us think about parallel problems; ConcX encourages us to imagine a parallel problem/program as resembling a flock of birds where each bird behaves and operates independently but also cooperates with the other birds in the flock. ConcX maps the various thread states into the natural behaviors of a bird, such as hatching (thread.start), napping (thread.sleep), death (thread.stop) and so on to make it  easier to think about and describe each thread’s actions. The mental process of breaking down a program’s tasks into a set of atomic actions that can be independently performed, one per bird, produces a scalable parallel solution to the program.

Set-up ConcX to Run Dining Philosophers Problem

The following screenshot shows the ConcX GUI used to select the types of birds and how to configure them. The GUI also allows the flock of birds or individual birds to be started or stopped while viewing in real time each bird’s activity on its progress bars.

To solve the Dining Philosophers problem, five “birds” were added (tabs #2 – #6) to create a new flock. Each bird gets its own tab so it can be independently configured. In this case, each bird was configured to “eat” two food types which are called Fork1Pod thru Fork5Pod. To set the table, a “waiter” bird was also added (who disappears after the table is set, which seems typical of waiters around the world).

Note that the ConcX download comes with all the bird and food objects needed to solve the Dining Philosophers problem and a variety of other parallel problems, such as Producer-Consumer, Calculating Pi in parallel, the Sleeping Barber problem, and more.

Solving the Dining Philosophers Problem

Click the Start All button (lower left corner) and a SwingWorker thread will be started for each bird in the flock. The philosophers (birds/threads) will all begin eating, as indicated by their individually growing Activity progress bars. After 30 seconds (their default lifetimes), the philosophers will all die, having reached the end of their (configurable) lifetimes, and their threads will all terminate. If you’re impatient, you can also click the Stop All button and they will all stop eating, terminating their individual threads.

There, that’s it. Problem solved. ConcX provided all the tools and the necessary framework to solve the Dining Philosophers problem. The philosophers all shared their forks and got to eat an approximately even number of times and none of them starved.

“Wait!!” I can almost hear you saying, “it can’t be that simple. Show me how that worked.”

Details of Dining Philosophers Solution

The following screenshot shows the progress of all the philosophers shortly after they were all started. Each of the progress bars was individually updated in real-time based on how many times their assigned philosopher ate. In this case, their progress bars are all about the same length, showing that they were sharing fairly equally, although Bacon was slightly less successful than the others at eating.

This solution to the Dining Philosophers problem works because each philosopher is a BasicBird in ConcX, which The Avian Lifecycleallowed them to all inherit the standard Avian lifecycle. As shown in the lifecycle diagram, once a BasicBird is hatched, it looks for its assigned type of food, digests any of its food that it found, stores the appropriate type of food and then takes a nap. It repeats this loop until it dies of old age, starves to death, or you push the stop button.

The lifecycle of a BasicBird just happens to map into the DIning Philosophers description: each philosopher trying to pick up two forks maps to the bird’s Looking for Food stage, eating spaghetti is performed during the Digest Food stage, putting down their forks happens during the Store Food stage, and thinking is done during the Nap stage (which is when I do all my best thinking).

Back to our example run where Bacon didn’t eat as often as the other philosophers. Note that each individual philosopher can be started using the Start Me button on his tab. If you’d like to see Bacon eat more, click on his tab (#3) and then click his Start Me button; he will start eating and be the only philosopher whose progress bar is growing. It will actually grow faster than before because he doesn’t have to share.

The following screenshot shows the results of what happens after Bacon was given the opportunity to make a pig of himself; his Activity tab progress bar is now longer than the other progress bars.

“Wait,” I can almost hear you saying, “growing progress bars don’t prove anything. I want to see some kind of evidence that something actually happened.”

To really get a feeling for what was happening during the run, on the GUI screen, click on the TupleTree tab at the upper right side of the screen. This will display the “history” of each ForkPod (see the following screenshot). The history of each food pod is always recorded so you can always review and analyze the events that happened to the food pod during it latest run.

In this enlarged portion of the screen, Fork2Pod is shared by Aristotle and Bacon; the TupleTree tab allows you to view the individual history of each food pod down to the millisecond. Note that Fork2Pod is shared evenly by Aristotle and Bacon at the beginning of the run but then about mid-screen Aristotle gets “greedy” and monopolizes Fork2Pod before sharing more evenly at the bottom of this screen. If you scroll down thru the history of Fork2Pod, you can view the time when Bacon is the only active Philosopher and the only one who uses the Fork2Pod, matching up with his progress bar being longer. You can also scroll down and view the histories of all the other food pods in the TupleTree.

TupleTree Discussion

The TupleTree in ConcX is a simplified version of a tuplespace, a concept originally developed for the Linda coordination language; a tuplespace is shared associative memory used as a repository for tuples of data. Linda was started in 1986 by Sudhir Ahuja, David Gelernter, Nicholas Carriero and others. Linda was the basis for several major products, including Sun’s JavaSpaces, IBM’s TSpaces, and many more in a variety of languages.

In ConcX, the TupleTree actively manages all shared objects (food pods). In this problem, when a philosopher attempts to pick up a fork, he is actually requesting a specific type of food pod from the TupleTree. If the TupleTree doesn’t contain the requested type of food pod, the philosopher gets nothing, which means he failed to pick up a fork and goes back to thinking (napping) for a random length of time.

However, if the TupleTree does contain the requested type of food pod (Fork1Pod, Fork4Pod, etc), the TupleTree removes the requested pod and gives it to the philosopher who requested it, who now has exclusive possession of the one and only instance of that food pod (fork). No other philosopher can access, change or modify that fork until the philosopher puts it down (stores it back in the TupleTree).

An interesting and extremely convenient by-product of the TupleTree is that the code for the BasicBird doesn’t contain parallel programming logic, such as mutexes, semaphores, locks, etc. All of the parallel programming code/logic has been factored out of the BasicBirds (users of data) and exists only in the TupleTree code (repository of data). The TupleTree has exclusive access to the data repository and manages all of the locking code so you don’t have to.

Preventing Deadlocks in ConcX

Back to the original problem description: after a philosopher has successfully picked up his first fork, he then attempts to pick up his second fork. This is the point where deadlocks typically can happen because the situation could arise where every philosopher will be holding their first fork and be unable to get their second fork because it is being held by their neighboring philosopher.

ConcX avoids deadlock situations by default because BasicBirds will give up after failing to get their second food (fork) a configurable number of attempts and put down the first fork, making it available to their neighbor. The reason that BasicBirds can be so well behaved is because they never block and wait for their fork. Instead, they only have to wait for a millisecond or so until the TupleTree responds with either the requested food pod or an empty food pod, meaning that the requested food pod wasn’t in the TupleTree. Receiving an empty food pod is a normal condition in ConcX; just like real life birds, BasicBirds that don’t find food just go on with their lifecycle and look for food the next time thru their lifecycle.

To demonstrate how the deadlock prevention works, view the lifecycle events for a particular bird by clicking on its numbered tab and then click on its individual History tab. In ConcX, each bird always records the events that happened to the bird during its latest run. The following screenshot shows a portion of Aristotle’s history from a recent run. The portion shown of its history begins with him looking for food and finding nothing so he takes a nap of 160ms. When he wakes up, he looks for food, gets the Fork1Pod and then tries three times before getting the Fork2Pod (his second fork). He then eats, digests and puts down his forks (stores them back in the TupleTree) so other philosophers can use them. After all that exertion, he takes a nap for 50ms (note the random duration of this nap) before looking for food again. And every event is timestamped so you can cross-check them with the food pod events on the TupleTree tab.

ConcX was specifically designed to be an interactive environment for experimenting: at this point you have enough information where you could download ConcX and start experimenting with the Dining Philosophers problem. For example, you could easily make the following experiments:

  • Change the lifetime of one or more birds to 100 seconds or 10 minutes or as long as desired

  • Reduce the “patience” (a configurable value) of one philosopher so he tries only once to get the second fork before giving up and putting down his first fork

  • Increase a philosopher’s patience to see how the results are changed if he tries 10 times to get the second fork instead of the default 5 times.

  • Reduce a philosopher’s Nap length to see what happens if he checks more frequently for available forks. Will his eagerness (greed) result in more frequent eating? How will a greedier philosopher affect his neighbors? What happens if the neighbors also become greedy?

These experiments and more can be run just by changing the philosophers’ configurations, clearing the previous activities/results and restarting the philosophers. Many more experiments can be run by adding more philosophers. For example:

  • Add a sixth philosopher without adding another fork. Just click the Add New Bird button, give it a name, select a type of bird, and then pick which forks to eat and store and restart the simulation.

  • Add five more philosophers (10 total) without adding more forks so each plate at the table has two philosophers who share the same two forks

  • What about a bigger table with 10 philosophers and 10 forks?

  • What is the maximum number of philosophers that can be fed without any of them starving and without also increasing the number of forks?

ConcX makes it easy to explore these possibilities and more, all without coding. And if you like the simulation or results, you can save them to a new flock file so you can reload and rerun the them later.

If you’re willing to do some coding, the possibilities are endless. For example, what about an opportunistic philosopher who will use any forks and will eat any time he can find two unused forks? What if one or more forks was bent or twisted so the philosophers also tried to trade their forks to get the better forks?

Background

Parallel programs take too long to develop. Multi-core machines have been the standard CPUs for most modern computers for almost a decade but most software struggles to make full and balanced use of those cores. If software could have kept up with hardware, it is likely that 16-core or 128-core CPUs would be the norm.

The Avian Computing Project began with the idea that if we can better visualize and describe how a parallel program should operate, we can reduce the length of time it takes to develop that parallel program. This insight lead to a search for a model that could naturally and intrinsically exhibit parallel behavior; a flock of birds was selected as the model for the project, although a school of fish or a hive of bees or pack of dogs could just as easily been selected.

These models allow us to easily think about the required actions of one individual and then just as easily think about the whole group and how the changes made to one could affect the whole group. Further, using a model from nature makes it easier to create mental images of the operation of individual threads because the various stages in a thread’s lifecycle can be mapped into easily understood stages in an animal’s lifecycle. Want proof? Try explaining how a multi-threaded program works to a 5-year old. Now try explaining how a flock of birds might do some useful actions to that same 5-year old and the child will understand.  

Contrast the Avian model with the standard way of writing parallel code where mutexes and locks get sprinkled throughout the code anywhere we suspect multiple threads might access some shared resource. Typically we can’t predict when the various threads will access the shared resources or when the system will interrupt a thread so all we can do is try to imagine all the possible conditions that would cause problems and then test and hope we find any actual failure conditions that we hadn’t anticipated.

ConcX was developed to provide an interactive environment that allows developers to leverage the Avian model (flock of birds) to easily represent individual members of a group and then experiment with how the individuals work together to accomplish the program’s goals in parallel. ConcX provides a variety of pre-built objects that can easily be customized to accomplish most any program requirements, streamlining and speeding up the job for developers. ConcX also structures and confines all locking and synchronizing to the shared TupleTree object, limiting any issues with inappropriate access of objects to just the TupleTree.

Using the code

The following code demonstrates how a new type of bird is created, which is typically done by extending BasicBird and then overriding the appropriate methods. In the Dining Philosophers problem, the Waiter is a special purpose bird whose only work is to put five forks onto the table (into the TupleTree). The complete code is listed below and requires only 47 lines (including comments and blank lines), only 38 after blank lines are removed, and only 24 lines when comments and blank lines are removed and stand-alone curly braces moved onto other lines.

public class Waiter extends BasicBird {

   /**
    * The waiter sets the table for the philosophers by putting forks
    * into the TupleTree
    */
   @Override
   public void findFood() { //overridden so will store the forks in TupleTree
       Fork1Pod f1 = new Fork1Pod(); //creates a new instance of a Fork1Pod
       f1.setEmpty(false);           //The ForkxPods are all predefined food
       mEatBeak1.storeItem(f1, this);//pods that are used as forks

       Fork2Pod f2 = new Fork2Pod();
       f2.setEmpty(false);
       mEatBeak1.storeItem(f2, this);

       Fork3Pod f3 = new Fork3Pod();
       f3.setEmpty(false);
       mEatBeak1.storeItem(f3, this);

       Fork4Pod f4 = new Fork4Pod();
       f4.setEmpty(false);
       mEatBeak1.storeItem(f4, this);

       Fork5Pod f5 = new Fork5Pod();
       f5.setEmpty(false);
       mEatBeak1.storeItem(f5, this);

       addToBirdHistory("Table is Set - Ready to Terminate", Level.INFO);
       setStopNow(true); //this thread can now be terminated
   }

   /**
    * This method does nothing and prevents the regular digestFood method
    * from making any changes to any of the forks.
    */
   @Override
   public void digestFood() { }

   /**
    * This method does nothing and prevents the regular storeFood method from
    * storing any additional forks.
    */
   @Override
   public void storeFood () { }

}

Creating new food pods (ForkxPod) is similarly simple because it extends the BasicPod and assigns it unique identifiers. The following is the complete code to create the Fork9Pod.

package com.avian.foods.philos;
import com.avian.foods.basefoods.BasicPod;

public class Fork9Pod extends BasicPod {

   /**
    * Default (and only) constructor for this food, it calls the BasicPod
    * constructor with reasonable values for its variables
    */
   public Fork9Pod()
       super("phl09","Fork9Pod"); //constructor params = treeID, desc
       this.setPodType("Fork9Pod");
   }
}

The BasicBird code contains numerous stub methods to provide convenient ways of customizing the behavior of any bird inherits from BasicBird. For example, methods such as beforeDigesting and afterStoring, can be overriden to get exactly the desired behavior without having to modify the BasicBird code.

To learn more about Avian Computing and about programming for ConcX, download Getting Started with Avian Computing – Exploring Parallel Programming with ConcX available at aviancomputing.net or from the Avian Computing pages at SourceForge.

Points of Interest

Perhaps one of the most interesting aspects of ConcX is the absence of a main() function that sets up and/or controls the threads as is typical of most parallel programs. This has some interesting ramifications:

  • The threads of the ConcX program can be individually started and stopped at will because they are not pre-defined or compiled into the main() function or some setup method. They run as independent threads and will continue to run as long as they meet their own personal criteria to continue running.
  • ConcX threads are very loosely coupled. They are unaware of other threads and never directly communicate with other threads; instead data chunks (food pods) are shared asynchronously through the TupleTree. This means that the code never has to be modified to add or delete threads.
  • Loosely coupled threads also allows flock behavior to be changed or rearranged simply by changing the foods that the birds eat and store.  For example, in the Dining Philosophers problem, instead of sharing forks with their neighbors, the philosophers could just as easily share forks with the philosophers seated across from them, just by changing the foods the philosophers eat.
  • Loosely coupled semi-autonomous threads raises the possibility that a market for special purpose bird objects could develop. For example, someone could develop and sell an Address Formatting bird that takes customer info as input and produced a properly formatted address as an output. 

Conclusion

Hopefully, this article has demonstrated that it is relatively simple to solve the Dining Philosophers problem using the objects, tools, and features of ConcX. Unlike most solutions to the Dining Philosopher problem, ConcX also provides the following:

  • real-time visual feedback about how successful each individual philosopher is at eating so it is easy to verify that none of them is starving and that they are sharing the forks

  • a truly asynchronous solution where each of the threads can be started/stopped independently without interrupting (or crashing) the program

  • event-recording tools to automatically capture runtime information so detailed post mortem analyses can be performed

  • Ability to re-configure the various philosophers and immediately re-run the Dining Philosophers to test the impact of the configuration changes

ConcX is a general purpose solution that was built to provide developers with an environment to help them think about their parallel programs. ConcX was designed to allow developers to focus on what their program needs to do instead of being preoccupied on how to write the code that safely shares data and resources.

History

May 7, 2018: Initial submittal

Salı, 29 Mayıs 2018 / Published in Uncategorized

Problem

I have shown how to uninstall SQL Server 2017 from an Ubuntu server in a past tip and in this tip I will show how to uninstall SQL tools (sqlcmd and bcp) on an Ubuntu server.

Solution

As you may know, Microsoft has extended its SQL Server support to Linux based operating systems starting with SQL Server 2017. We have to install SQL Server and SQL Tools (sqlcmd and bcp) separately using two different install packages. SQL Tools contains command-line utilities, Microsoft ODBC drivers, and their dependencies. The package name of the SQL Tools is mssql-tools. This package contains sqlcmd and bcp utilities as well. We use the sqlcmd utility to connect to a SQL Server instance and execute queries, where the bcp utility is used for bulk import and export operations.

If we need to uninstall these tools from any Linux system, we need to uninstall it separately from SQL Server. The uninstall of SQL Server 2017 will not remove SQL tools from your Linux database server.

I have also written multiple tips on how to install SQL Server 2017 on different Linux flavors. Have a look at the “Next Steps” section to access all of those tips.

In this tip we will cover the steps to remove the sqlcmd and bcp utilities from an Ubuntu Linux machine.

Uninstall SQL Server 2017 Running on Ubuntu

I have installed SQL Server 2017 along with SQL Tools (sqlcmd and bcp) on this machine for some project work. Now, I have to uninstall it and clean the Ubuntu machine. I have already uninstalled SQL Server 2017 in my last tip and now we will cover how to remove SQL Tools.

Step 1:

You might have a requirement to remove only SQL Tools from your system. It may not be mandatory to remove your SQL Server instance running on the Linux sever before removing SQL Tools. But, generally we remove SQL Tools as part of the full SQL Server removal from any machine. If you have already uninstalled SQL Server from your Ubuntu server then the next step is to remove SQL Tools to clean your machine. The first step is to connect to your Linux database server by entering your login name and password. Once connected to your Ubuntu Linux server, then we can go ahead and verify whether SQL Tools are installed or your machine or not by executing the below command.

#Type sqlcmd on your Linux terminal and press enter
sqlcmd			

By trying to run sqlcmd command on your Linux terminal, we can see if your Linux system recognizes this command which means SQL Tools are installed. You can see this in the below screenshot that this utility has been recognized by the Ubuntu Linux server.

Step 2:

Now you can check whether a SQL Server instance is installed on this machine or if it has already been removed. You can check the SQL Server service by running the below command.

#Run below command to check SQL Server Instance running on this Linux server.
systemctl status mssql-server 			

You can see there is no SQL Server instance on this server in the below screenshot.

Step 3:

Our next step would be to remove SQL Tools (sqlcmd and bcp) from this server. We will use the apt-get command to remove these utilities. The apt-get is the command-line tool for working with core libraries to handle the installation and removal of software on Linux distributions like Ubuntu, Debian, etc.

Here we will use the apt-get remove command to perform this uninstall on this machine. Run the below commands to remove the SQL Tools from your Ubuntu server.

#Run below command to remove SQL Tools
sudo apt-get remove mssql-tools			

This command will read the package and show you what components will be removed. This command will ask you to enter Y to proceed with the removal and enter n to quit this uninstall. I entered Y to proceed. You can see the SQL Tools have been uninstalled from this server in the below image.

Step 4:

Now, we can validate whether SQL Tools (sqlcmd and bcp) have been uninstalled from this machine. We can get this by running sqlcmd command from your Linux terminal as we did in Step 1. We can see that Ubuntu does not recognize this command anymore and says, “No such file or directory”.

Step 5:

Another way to validate this uninstall is by removing the SQL Tools package again by executing the same commands we ran in Step 3. You can see the command apt-get is not able to locate the SQL Tools package on the server because we have already uninstalled it.

Once the tools are uninstalled, you will not be able to use sqlcmd and bcp utilities. You will not be able to connect to SQL Server from local Ubuntu server since you need sqlcmd to connect to the database from the local server to perform any query executions.

Next Steps

Last Update:
2018-04-11

About the author

Manvendra Singh has over 5 years of experience with SQL Server and has focused on Database Mirroring, Replication, Log Shipping, etc.

View all my tips

Salı, 29 Mayıs 2018 / Published in Uncategorized

Problem

Some technical problems do not lend themselves to storing and retrieving data from a predefined table.  The popularity of NoSQL databases started with companies such as Facebook, Google and Amazon when working with big data problems and/or real time web applications.  This set of technology is coined “Not Only SQL” since they might support a SQL syntax for retrieving data.

While there are many NoSQL database systems, most systems can be placed into one of four widely implemented data stores. 

A key value store uses the concept of an associative array to save and retrieve data.  The Webster dictionary is the physical implementation of this concept.

A document store encapsulates all the data for an entity (document) using a popular encoding format such as XML or JSON.  This technology is popular with applications that have a variety documents with different attributes.

A graph store contains a bunch of elements with a finite number of connections between them.  Data such as social relationships, road maps and/or network topologies can be easily represented with this technology. 

A column store saves the data using columns instead of rows.  This means that columns with discrete values can be highly compressed.

How can we use a document store technology in Azure to save and retrieve data?

Solution

Azure Cosmos database is Microsoft’s new globally distributed, multi-model database service.  Right now, there is support for five different application programming interfaces:  key value store (table), graph store (gremlin), document store (SQL), Mongo database (documents) and Casandra database (wide column).  The highlighted words represent the API option to choose when creating the database account.  Please see books on line for details.  It is interesting to note that many changes happened to Cosmos database last year.

While there is support for four different application programming interfaces, today we are going to use the SQL API to manage JSON documents.

Business Problem

There are many financial companies that invest in the stock market via mutual funds. Our fictitious company named Big Jon Investments is such a company.  They currently invest in the S&P 500 mutual fund but are thinking of investing in individual funds starting in 2018.

The investments department will need historical data to make informed choices when picking stocks.  Our boss has asked us to load summarized daily trading data for the 505 stocks that make up the list into a key value store.  We have five years’ worth of data stored in comma separated files.  Each file has the stock symbol and year as part of the name.

How can we insert, update, delete and select records from Azure Cosmos DB using the SQL API?

Saving Stock Data in JSON format

In a previous article, I talked about the how to use Power Shell to load Azure Table Storage with stock data.  There was 2,525 comma separated value (CSV) files that represented five years’ worth of S&P 500 stock data.

Almost all of the services published in Azure use the REST API. Roy Fielding proposed Representational State Transfer (REST) as an architectural approach to designing web services in the year 2000.

With these facts, we want to transform each of the records stored in the CSV file into a document using the JSON format.  I am enclosing the PowerShell program that I used to make this conversion happen.

The image shown above is taken from Windows Explorer.  It shows that 505 folders were created, one for each S&P 500 stock symbol.  There are 627,326 JSON files or 232 MB of data to load into Azure Cosmos DB.

The above image shows the JSON files stored in the MSFT directory.  This is the stock symbol for the company named Microsoft Inc.

The above collage shows the MSFT directory, the size of one JSON data file, and the contents of one JSON data file.

There are four additional fields that have been added to the document for lineage.  First, each document has to have a unique key. The document identifier (docid) is comprised of the stock symbol and the trading date.  Second, each document should have a version number (docver).  Since the schema is fluid in a NoSQL data store, we need to have governance that keeps track of the differences between version 1.1 and version 1.2.  Last, it might be of interest to record who modified the record (moduser) and when was the record modified (moddate).

One decision that you need to make is how to deal with dates?  They can be stored as a date string or as Unix epoch values. Read this article by John Macintyre, a partner manager at Microsoft, on this subject.  I chose to store dates as searchable strings.

In a nutshell, you might want to add additional fields to the JSON document to satisfy your business requirements.

Azure Cosmos DB Overview

Before we create our first Azure Cosmos database, it is warranted to have an architectural overview of the document store technology.  I am going to state the English old idiom.  A picture is worth a thousand words.

The above diagram taken from MSDN shows each of the components that maybe be used when designing your document store.  I am going to call out the objects we need to define before we can load our stock data into the document store.

The database account is where you choose the API that you want to work with.  In our case, we want to use the SQL API.  Each account can have one or more logical databases.  This is where users and permissions are defined.  Each database can have one or more logical collections.  This is where documents are group together.  Resources units (RU) are defined at this level.  Last but not least, collections can contain zero or more documents.  The documents can be either homogenous or heterogeneous in nature.

Now that we have a high-level understanding of the database system, let us design our first document store using Azure Cosmos DB.

Creating Azure Objects in Azure Cosmos DB

This article assumes that you have an existing Azure subscription. If you do not, Microsoft offers new customers a 30-day trial subscription with a $200-dollar spending limit. Use this link to sign up now.

Once you logged into the Azure Portal, choose create a resource, databases, and Azure Cosmos DB.  The following screen shot shows the dialog box to create a new database account.

I am using a naming convention for the objects that I define in the Azure Portal.  The database account named cdba4tips18 will be deployed in the east us 2 region within the rg4tips18 resource group.  Of course, we want to use the SQL API to manage our data.  Please note that I choose to pin this object to the dashboard as a short cut.

The image below shows the deployment of the Azure Cosmos DB database.  Because this service is globally distributed, an image of the world is shown with all the regions shown as hexagons.  Our primary service or database is located in the us east 2 region.  Last but not least, the URL for our service is displayed in the top right corner.  We will talk about replicating data in a future article.

Before we can add JSON documents to the service, we need to define a database and collection.  Click the data explore button highlighted in blue to defined these objects now.

I am choosing to name the database db4BigJonsLlc and the name the collection col4CompanyData.  There are two schools of thought when naming objects.  The client, project and environment can be the prefix of the name so that all related objects are shown together in the portal.  On the other hand, the type of object can be used as the prefix so that all similar objects are grouped together.  As long as your company has a naming convention and the developers stick to the rules, your Azure objects should be grouped accordingly.

By default, a collection without a partition key is limited to 10 GB of data.  Since partitions and replication are out of scope for this introductory article, we will talk about them in the future.  I am choosing to dial down the resource units for the collection to the minimum. That means, a 10 GB collection will cost less than $25 dollars per month. Defining a unique key for the document is a requirement.  We will select the document identifier (docid) field we added to the stock data.

Click the okay button to deploy the database and collection.

Three Explorers

The Azure portal has three explorers that can be used to manipulate our newly created collection.  The document explorer allows you to add, update and delete documents from a collection.  The query explorer allows you to select document data from the collection using a SQL like syntax.  To round out our tools, the script explorer allows you to define JavaScript objects to support advanced functionality.

We will explorer each of these tools in the upcoming sections.

Manually Adding Data

The JSON document store would be of no use to our users if there is no data.  Click the document explore and the upload button.  Select the first seven documents related to stock symbol A or Agilent Technologies, Inc.  Click the upload button to complete the action.

You should now see seven globally unique identifiers (GUIDS) related to the documents you just uploaded.  Double click the first document to open it in a web editor.

The image below shows the JSON document related to the data for stock symbol A on January 10th, 2013.  The Cosmos database system automatically adds the id to the document for internal use.

The data explorer can be used to load the collection with a limited number of documents.  At most, you can load 100 files at a time.  There must be a better way to load our half a million JSON documents to satisfy our business requirements?

Selecting Document Data

The query explorer is used to craft SQL statements to retrieve data from our JSON document store.  For instance, a financial analyst at Big Jon’s Investments asked us to return an array of unique identifiers for 7 documents we recently loaded.

The Cosmos database supports a wide variety of SQL statements for the document store API.  I will be talking about the various syntax you can used to retrieve data from this service in future articles.   The statement below returns the docid attribute for the collection c which is defined as col4CompanyData.

The image below shows the results of the SQL query.

To recap, the document store in Azure Cosmos DB supports querying documents using a familiar SQL (Structured Query Language) like grammar over hierarchical JSON documents without requiring explicit schema or creation of secondary indexes.

Server-Side Programming

JavaScript is the chosen language to extended your design with Stored Procedures, pre and post document creation triggers and user defined functions.  Again, these concepts are out of scope for an introductory article.  However, I did want to mention that they do exist.

The above image shows the script explore in the Azure Portal.  This is where you can deploy your user defined JavaScript code for stored procedures, triggers and functions.

Data Migration for Azure Cosmos DB

The current data explorer in the Azure portal only allows 100 documents to be uploaded per session.  We would have to repeat this process more than 6,000 times to load the S&P 500 historical stock JSON documents.  This is totally unfeasible.  How can we load this data set as an easy and repeatable process?

Microsoft has supplied the administrator with a data migration tool for Azure Cosmos DB.  The tool can be downloaded from this web page.  This tool has not been updated in a while since it still references Document DB.  On the other hand, the tool works just fine.

The first step is to download and unzip the files into a directory.  Next, launch the dtui.exe program from Windows Explorer.

The second step is to define the source information.  Choose add folder with the recursive sub-directory option.  The image below shows the “C:\JSON DATA\” directory containing our source documents.

The third step is to define the target information.  We need to retrieve this information from the Azure Portal.  The image below shows the URL and PRIMARY KEY for our Cosmos Database account named cdba4tips18.

Capture and save this information for the target dialog box.

The image below shows the target dialog box.  The connection string is comprised of the URL, the account key and the database name.  We also need to supply the collection name, the id field and the collection throughput RU.

Click the advanced options to choose additional import settings.  I have decided to connect using direct TCP, use 25 parallel requests for the import, update any existing documents and save dates as a string format.  Last but not least, I will try the operation 30 times before marking it as a failure.

The fourth step is to define where to store error information, and how often do you get notified about the update process.  Error logging is very important when determining the root cause of a failed insert.

The fifth and last step is to confirm the import settings before execution.  Since I am happy with my choices, I am going to click the import button.

The image below shows the summary information at the end of execution.  To display some error information, I did start and stop the process one time.  This resulted in 839 records already being stored as documents.  If we take a look at the error file, it states that there was a “unique index constraint violation”.

There was a total of 626,487 records imported in 18,110 seconds.  That is roughly 34.59 records per second for a duration of 5 hours, 1 minute and 50 seconds.

The Microsoft data migration tool is one easy way to load a bunch of documents into the document store.

Checking the Migration to Azure Cosmos DB

Today, we just inserted 627,326 documents into our collection named col4CompanyData which resides in our database named db4BigJonsLlc.  Of course, both objects are associated with our database account named cdba4tips18.

Let us write a query to retrieve the number of trading days in which we have data for the stock symbol “MSFT”.

SELECT COUNT(c.docid) as total
FROM c
WHERE c.symbol = ‘MSFT’ 

The above image returns a document count of 1,259 which is the correct answer.

Now, let us remove the WHERE clause to COUNT all the documents in the store. Okay, the number 29,535 is the wrong answer.  What is going on here?

Unlike a relational database, the collection is throttled at the number of RUs we defined.  For collections without a partition key, we can scale this number from 400 to 10,000. If we look at the details of the query execution, we can see that we used around 2314 RU’s and there is more data to be retrieved to arrive at the correct result.

In conclusion, queries might complete with partial results.  I leave it up to you to scale up the resource units (RU’s) for the collection and re-execute the query.  Maybe it will finish with the correct result.

Azure Cosmos DB Summary

Today, we talked about how a NoSQL database can be categorized as either a key value store, a document store, a graph store or a column store.

Azure Cosmos database is Microsoft’s new globally distributed, multi-model database service.  Right now, there is support for five different application programming interfaces:  key value store (table), graph store (gremlin), document store (SQL), Mongo database (documents) and Casandra database (wide column). 

The S&P 500 historical data set contains data on roughly 505 stocks over the past five years. A PowerShell program was designed to convert the 2,525 comma separated value (CSV) files into 627,326 JSON files or 232 MB of data.

The business problem we solved today was inserting and selecting data from the Azure Cosmos DB database using the Azure Portal.  Unfortunately, the portal is not designed to import large amounts of data.  Fortunately, Microsoft has supplied the administrator with a migration tool.  This tool was used to load the data into Azure in about 5 hours.

Quality assurance is part of being an excellent administrator.  A simple query to return the total number of trading days for the MSFT stock was designed and executed.  The correct number of records was returned.

Next, the same query was modified to count the total number of records loaded into the database.  Since this summary information is not part of the inverted columnar tree index, a full search was executed.  A partial result was returned by the portal since the query was throttled by the resource unit governor.

In short, there is a lot more to learn about this NoSQL document store.  I will be deep diving into some of the topics listed below in the future.

Next Steps
  • Exploring the query syntax available with the SQL API
  • Understanding partition and replication for distributed data
  • Exploring the different consistency models in the SQL API
  • Understanding server-side programming for the SQL API
  • Understanding the data migration tools for Azure Cosmos database
  • The different application SDK’s available with the SQL API
  • Using table storage API with Azure Cosmos database
  • Exploring graph storage API with Azure Cosmos database

Last Update:
2018-04-12

About the author

John Miner is a Data Architect at Blue Metal helping corporations solve their business needs with various data platform solutions.

View all my tips

Salı, 29 Mayıs 2018 / Published in Uncategorized

Problem

SQL Server can store a large volume of data in relational formats which is great for the business, but business users and developers also have needs to store documentation and information related to the SQL Server objects. One way to do this is to use Extended Properties which allows you to save information about the objects such as what it’s for, specific formats like phone format, date format, description of objects, URLs, website links and so on. In this tip we will show how we can add Extended Properties.

Solution

Extended Properties is a unique feature in SQL Server to store more information about database objects.  In this article we will see how to:

  1. Add, Update and Drop Extended Properties.
  2. Extract the Extended Properties from sys.objects and sys.extended_properties tables.
  3. How to use function FN_LISTEXTENDEDPROPERTY() to extract Extended Properties.

Extended Properties can be created for the below database objects, but in this tip, we are going to focus on column level Extended Properties.

  • Database
  • Stored Procedures
  • User-defined Functions
  • Table
  • Table Column
  • Table Index
  • Views
  • Rules
  • Triggers
  • Constraints

Extended Properties can be used to:

  • Specify a caption for a table, view, or column.
  • Specify a display mask for a column.
  • Display a format of a column, define edit mask for a date column, define number of decimals, etc.
  • Specify formatting rules for displaying the data in a column.
  • Describe a specific database objects for all users.

Example of Creating Extended Properties

Let’s create a table that has two columns “sno” and “myName”.

IF OBJECT_ID ('MyTest','U') IS NOT NULL
    DROP TABLE MyTest;
GO

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE MyTest (sno int, myName char (20))
GO	

To look at the Extended Properties for a column, in SSMS expand Tables, find the table we created and then expand the Columns. Then right click on column "sno" and select Properties and go to the Extended Properties page.

The below screenshot shows the Extended Properties for column "sno" is empty after the table is created. If you want to add an Extended Property you can just type in the Name and Value on the screen below.  I will also show how to do this with T-SQL.

Add SQL Server Extended Property with sp_addextendedproperty

We can also use sp_addextendedproperty to add an Extended Property. The below stored procedure needs to be executed with parameters as follows.

exec sp_addextendedproperty  
     @name = N'SNO' 
    ,@value = N'Testing entry for Extended Property' 
    ,@level0type = N'Schema', @level0name = 'dbo' 
    ,@level1type = N'Table',  @level1name = 'mytest' 
    ,@level2type = N'Column', @level2name = 'sno'
go			

A few parameters are required to execute sp_addextendedproperty.

  • @name is ‘SNO’ in our case. This cannot be null. This is the name of the Extended Property.
  • @value is the value or description of the property and it cannot exceed 7500 bytes.
  • @level0type in our case ‘Schema’ and @level0name is the value is set as ‘dbo’ as the value
  • @level1type in our case ‘Table’ and @level1name is ‘mytest’
  • @level2type in our case ‘Column’ and @level2name is ‘sno’

The below screen shows the added Extended Property using sp_addextendedproperty. It shows the name and value for the Extended Property.

Querying SQL Server Extended Properties

The sp_addextendedproperty will create rows in the sys.extended_properties table, by storing data in this table SQL Server has the ability to fetch the data as per the request. In many documentation automation projects, these tables can be queried and the data can be used for documentation purposes.

Below is the query to extract data about the ‘MyTest’ table.  We can see the object_id value for the table.

select * 
from sys.tables  
where name = 'MyTest'			

Below we can query sys.extended_properties to get more information. We can also see the major_id matches the object_id above.

select * 
from sys.extended_properties 
where NAME = 'SNO'		

If we run a SQL Server Trace we can capture what SQL Server uses in SSMS, we find the following query is used by SSMS to pull the data for column level Extended Properties.

exec sp_executesql N'SELECT
p.name AS [Name],
CAST(p.value AS sql_variant) AS [Value]
FROM
sys.tables AS tbl
INNER JOIN sys.all_columns AS clmns ON clmns.object_id=tbl.object_id
INNER JOIN sys.extended_properties AS p ON p.major_id=tbl.object_id AND p.minor_id=clmns.column_id AND p.class=1
WHERE
([email protected]_msparam_0)and(([email protected]_msparam_1)and(([email protected]_msparam_2 and SCHEMA_NAME(tbl.schema_id)[email protected]_msparam_3)))
OPTION (FORCE ORDER)
',N'@_msparam_0 nvarchar(4000),@_msparam_1 nvarchar(4000),@_msparam_2 nvarchar(4000),@_msparam_3 nvarchar(4000)',@_msparam_0=N'SNO',
@_msparam_1=N'sno',@_msparam_2=N'MyTest',@_msparam_3=N'dbo'

We can simplify this and use the following query to get the Extended Property for the column.

SELECT
   SCHEMA_NAME(tbl.schema_id) AS SchemaName,	
   tbl.name AS TableName, 
   clmns.name AS ColumnName,
   p.name AS ExtendedPropertyName,
   CAST(p.value AS sql_variant) AS ExtendedPropertyValue
FROM
   sys.tables AS tbl
   INNER JOIN sys.all_columns AS clmns ON clmns.object_id=tbl.object_id
   INNER JOIN sys.extended_properties AS p ON p.major_id=tbl.object_id AND p.minor_id=clmns.column_id AND p.class=1
WHERE
   SCHEMA_NAME(tbl.schema_id)='dbo'
   and tbl.name='MyTest' 
   and clmns.name='sno'
   and p.name='SNO'
      

Here is the output.

Get all column level Extended Properties

If we want to get column level Extended Properties for all columns in the database, we could run the following.

SELECT
   SCHEMA_NAME(tbl.schema_id) AS SchemaName,	
   tbl.name AS TableName, 
   clmns.name AS ColumnName,
   p.name AS ExtendedPropertyName,
   CAST(p.value AS sql_variant) AS ExtendedPropertyValue
FROM
   sys.tables AS tbl
   INNER JOIN sys.all_columns AS clmns ON clmns.object_id=tbl.object_id
   INNER JOIN sys.extended_properties AS p ON p.major_id=tbl.object_id AND p.minor_id=clmns.column_id AND p.class=1
      

Here are some other example queries.

Get all database level Extended Properties

SELECT
   DB_NAME() AS DatabaseName,
   p.name AS ExtendedPropertyName,
   p.value AS ExtendedPropertyValue
FROM
   sys.extended_properties AS p
WHERE
   p.major_id=0 
   AND p.minor_id=0 
   AND p.class=0
ORDER BY
   [Name] ASC

Get all table level Extended Properties

SELECT
   SCHEMA_NAME(tbl.schema_id) AS SchemaName,	
   tbl.name AS TableName, 
   p.name AS ExtendedPropertyName,
   CAST(p.value AS sql_variant) AS ExtendedPropertyValue
FROM
   sys.tables AS tbl
   INNER JOIN sys.extended_properties AS p ON p.major_id=tbl.object_id AND p.minor_id=0 AND p.class=1

Get all stored procedure level Extended Properties

SELECT
   SCHEMA_NAME(sp.schema_id) AS SchemaName,	
   sp.name AS SPName, 
   p.name AS ExtendedPropertyName,
   CAST(p.value AS sql_variant) AS ExtendedPropertyValue
FROM
   sys.all_objects AS sp
   INNER JOIN sys.extended_properties AS p ON p.major_id=sp.object_id AND p.minor_id=0 AND p.class=1
WHERE
   sp.type = 'P' OR sp.type = 'RF' OR sp.type= 'PC'

Get Extended Property Using fn_listextendedproperty

We can also use the fn_listextendedproperty function to get a list of Extended Properties. The function returns objtype, objname, name with datatype sysname and value as sql_varient. NULL can be used as a parameter for the object name to return multiple extended properties, but it is not as straight forward as you would think it should be, to use this function.

The function can be found in the master database under Programmability > Functions > System Functions.

Execute the below query to get the Extended Property.

SELECT *
FROM ::fn_listextendedproperty ('SNO', 'Schema', 'dbo', 'Table', 'mytest', 'Column', 'sno')	

In the above query we notice that the first parameter is the property_name “SNO” and the other 6 parameters are object level and object type from level 0 to 2 in our example.  Below is the output:

Update an Extended Property with sp_updateextendedproperty

We can use sp_updateextendedproperty stored procedure to update the value of existing extended property.

exec sp_updateextendedproperty
     @name = N'SNO' 
    ,@value = 'SNO ID must be unique.' 
    ,@level0type = N'Schema', @level0name = 'dbo' 
    ,@level1type = N'Table',  @level1name = 'mytest'
    ,@level2type = N'Column', @level2name = 'sno'
GO			

The update stored procedure is used to update the value in the extended property as “SNO ID must be unique”. The update stored procedure is similar to the add stored procedure which accepts the same parameters and its retrieves the row based on the Name property which is ‘SNO’ in our example.

The below screen print shows the updated text in our example.

We can query the data again using the function to show the change.

SELECT *
FROM ::fn_listextendedproperty ('SNO','Schema', 'dbo', 'Table', 'mytest','Column', 'sno')	

Drop Extended Property with sp_dropextendedproperty

The sp_dropextendedproperty removes an extended property from the database. The following shows how to remove an entry.

exec sp_dropextendedproperty @name=N'SNO'
  ,@level0type = N'Schema', @level0name = 'dbo' 
  ,@level1type = N'Table',  @level1name = 'mytest' 
  ,@level2type = N'Column', @level2name = 'sno'
go			

We can query the data again using the function to show the entry has been removed.

SELECT * 
FROM ::fn_listextendedproperty ('SNO','Schema', 'dbo', 'Table', 'mytest','Column', 'sno')
			

Conclusion

We have seen the how to add, update and delete Extended Properties in SQL Server. Also, we saw how fn_listextendedproperty can be used to query the available Extended Properties.

Extended properties are a useful feature in SQL Server which can be used for documentation and content purposes. The properties can be updated for tables, views, triggers and so on. Developers can use this feature for extensive database objects which can be used for reference for many SQL Server objects.

Next Steps
  • Details about the SQL Server System Catalog Views are available here.

Last Update:
2018-04-13

About the author

Jayendra is a Project Leader with many years of IT experience. He has strong knowledge in software development and project management.

View all my tips

TOP